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Executive Summary 

This technology evaluation report documents the findings and recommendations of the 
Design for Safety Program’s PRACA Enhancement Pilot Study of the Space Shuttle 
Program’s (SSP’s) Problem Reporting and Corrective Action (PRACA) System. A team 
at NASA Ames Research Center (ARC) performed this Study. This Study was initiated 
as a follow-on to the NASA chartered Shuttle Independent Assessment Team (SI ) 
review (performed in the Fall of 1999) which identified deficiencies in the current 
PRACA implementation. The Pilot Study was launched with an initial qualitative 
assessment and technical review performed during January ^OOO with the quantitative 
formal Study (the subject of this report) started in March 2000. The goal of the PRACA 
Enhancement Pilot Study is to evaluate and quantify the technical aspects of the SSP 
PRACA systems and recommend enhancements to address deficiencies an in 

preparation for future system upgrades. 

The PRACA systems and their supporting infrastructure (used to report discrepancies 
non-conformances, problems, track engineering dispositions, corrective actions and 
provide data for trend analysis and reporting) are an essential tool for managing Shuttle 
safety and readiness for flight. The NASA Johnson Space Center (JSC) PRACA 
Evaluation Team (PET) was created to address the findings and recommendations from 
the SI AT, the initial ARC assessment comments, and other SSP sponsored PRAC 
audits and reviews. The PET was established by SSP Review Control Board Action 
S060341R5(3-1). The PRACA Enhancement Pilot Study was coordinated with the JSC 
PET, and as part of a new NASA Initiative - the Design for Safety Program (DfS). 

This Study report documents and provides a technical evaluation of the existing and 
currently operating SSP PRACA systems and quantifies technical and architectural 
issues This evaluation then generalizes the technical findings and recommends 
enhancements to improve this critical NASA distributed information system. The 
following four areas were assessed for each of the SSP PRACA systems. 

• User Interface 

• Database and Data Management 

• Network and System Architecture 

• Problem Reporting Work Processes 

A key element of the continuing success of the Space Shuttle Program and the operation 
of the multiple PRACA systems has been the dedicated and enthusiastic staff of NAS 
»d its contractor team. The progress of this Study was greatly aided by the tremendously 
dedicated and hard working individuals supporting the Space Shuttle Program. Everyone 
we spoke to through the course of this Study was highly cooperative and willing to ass 
us in completing this Pilot Study report and to ensure a continued safe Space Shuttle 

System. 

The overall assessment of the existing PRACA systems is that they are inefficient and 
potentially vulnerable to data loss and input error. The current approaches do not scale or 
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adapt easily to changes. The expert knowledge that is required to utilize the PRACA 
systems is not captured or documented. Overall, the exiting PRACA system is 
incapable of supporting Program-level risk assessment. 

Findings 

This Study’s findings and recommendations support and extend those of the SIAT report. 
Based upon this overall assessment, the Study presents the following general findings: 

1. There is a general vagueness in the definition of PRACA, its intended use, customers, 
and users that allows the inaccurate impression that the PRACA data systems meet 
the SSP’s needs. The current PRACA implementation shields the SSP from the many 
deficiencies and weaknesses in ne PRACA systems through th use >f highly skilled 
human resources and external data sources upon which the SSI nd PRACA depend. 

2 . The current PRACA systems’ technological basis and implementations are 
insufficient to fulfill Program-level data mining and safety assessment, and to support 
a Program level of safety, reliability, quality and mission assurance (SRQ&MA) 
analysis. 

User Interface and Trending 

• User interfaces for all systems are inconsistent or non-existent. The interfaces 
assume a specific user type that is different across all of the element PRACA 
systems. This prevents simple navigation across all PRACA systems 

• Trending and Analysis is often performed using non-PR AC A systems or only 
accessing PRACA data as a portion of the data used. This has allowed PRACA to 
evolve with insufficient data for statistical trending and insufficient supporting 
information for identifying data relationships in support of data mining. 

• The SSP office and its Project-level management currently meets the necessary 
condition of having enough problem reporting data and insight by relying on a set 
of domain experts possessing extensive knowledge of the Shuttle subsystems and 
the PRACA data. These experts have access to additional non-PRACA data 
sources and produce consolidated reports and summaries for the SSP from which 
the SSP performs its tasks and formulates decisions. This is a time-consuming, 
labor-intensive, and workforce skill-level dependent process. This precludes 
sustainable consistent processes. 

• For trending and risk analyses, additional data are required to produce results of 
statistical significance. These data are generated by grouping, or from augmenting 
databases. Some data are “scrubbed” during reporting to present a “correct” 
result. Additional data are scrubbed by staff distant from data acquisition and 
intimate knowledge of the possible reason for the questionable entry. 

Database and Data Management 

• Database schema and data fields collected are incomplete, inconsistent and not 
structured for data analysis. 

• The different disciplines’ definitions of PRACA data field values yield different 
interpretations across the PRACA systems. 



• The United Space Alliance’s ADAM data warehouse is a unified store for 
PRACA data and some associated information but provides no mapping across 
the various schemas or data field-naming conventions. As a result, queries across 
the PRACA systems are via an undocumented interface and cannot be extended. 

This limits any future Program-wide PRACA data mining. 

. PRACA is dependent on paper records, printed PRACA forms, and includes 
instances of re-keyed data. This raises concerns about data transcription errors and 

data integrity. . 

• There are multiple data sources on maintenance, repair, corrective actions an 

engineering dispositions (corrective action reports, hazard reports, engineering 
databases, expert knowledge, etc.) which are used to generate reports to the SSP 
but are not cross indexed with PRACA data. This means that a global picture of 
Shuttle health is not easily accessible. 


System and Network Architecture 

• Computer system hardware implementations supporting PRACA data 
management are all unique (some are 20+ years old) and not managed as a 


Program resource nnAP . 

Network access to relevant data is difficult due to the location of the PRACA 

systems throughout the NASA and contractor networks 

Security is incomplete and inconsistent across the implementations. There are 
inconsistent authorization and authentication processes and no encryption of data 
during network transfer. 

There is no SSP security policy for system implementation and data protection. 


3. A unified “PRACA System” as an organizational/programmatic entity does not exist. 

• The creators and element level managers of the PRACA systems do not view their 
PRACA systems as Program-wide resources and they have not been required to 

do so by the NSTS 08126 Revision G document. 

• The WebPCASS and ADAM/IPAS projects under Randy Segert at KSC are 
consolidating access to the data in many of the PRACA and related systems. 
However, this system is not being designed for the management or analysis of that 

data. . , 

• The element-only focus results in systems that are not useful for Program-wi e 

assessments and data analysis. Each system is unique and engineered for element 

use only with SSP Office use as an afterthought. 

• The use of the PRACA data as a Program resource to assess the Program-wide 
safety and risk of the Shuttle is a laborious and time intensive effort taking man- 
weeks to man-months. 


4. The motivation and requirements for the SSP “PRACA System” and its procedures 
and processes are unknown to the majority of the data providers and collectors. 

• The collectors of PRACA data are largely unaware of die value and potential use 
of the data gathered. Only a particular subset of data is “known to be desirable 
for any given element level PRACA system. 
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• To a large degree, collecting supporting PRACA data (e.g.. data needed by the 
SSP for problem background and broad problem documentation to support data 
mining) are not consistent with efficient workflow and are seen as burdensome 
requirements. 

• Element level training of the use of PRACA data is incomplete. The relative 
importance in the quality of the data being gathered is not understood. 

The JSC PET has rewritten and enhanced the NSTS 08126 document to a Revision H 
based upon the SIAT concerns and the initial ARC comments. This revision of NSTS 
08126 better reflects desired scope and global functions desired of the SSP’s PRACA 
system and clarifies the requirements for PRACA systems. This revision is a necessary 
step in the enhancement of the SSP Pi ACA systems. It is expected to be approved in the 
summer of 2000 by the Space Shuttle Program Review Control Board. We believe that a 
further clarification of PRACA requirements is still required. 

Recommendations 

The following recommendations are made to address the above findings, improve the 
quality, and enhance the use of the SSP PRACA systems: 

1. The SSP should clearly define PRACA, its intended use, customers, and users. This 
should include the operational scenarios and allowable data sources. The SSP should 
avoid overdependence on domain experts for data analysis and trend report 
generation. 

• Clearly identify (list) the Program-level PRACA tasks from a Program-wide 
perspective. 

• Establish requirements for a “PRACA System” that performs SSP level PRACA 
tasks (data retrieval, mining and trending needs). This action should be performed 
without consideration of current PRACA capabilities. 

• Design a “PRACA System” that satisfies these requirements. 

• Either a) Implement this new system or b) Initiate a modernization activity to 
upgrade the current PRACA systems and designs to satisfy the requirements. 

• Enhance the existing WebPCASS proposal based upon the above decisions. 

• Establish a plan for PRACA system evolution that will enable the development of 
a future Safety and Risk Prognostics capability. 

2. Develop and enhance the technical foundations upon which the PRACA Systems 
have been built. This is critical to enable the creation of a Program-level PRACA 
system capable of supporting the necessary breadth and depth of SRQ&MA analysis. 

• Enhance the ADAM data warehouse to become a central access point for 
Program-level SRQ&MA analysis on PRACA data. 

- Develop consistent database schema and structure, and common data field 
naming conventions and definitions. 

— Schema and structure should be designed to support SSP reporting, trending 
and data mining applications as well as to support the Project/Element 
workflow management. 
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- Schema and structure should be well documented to preclude data 
interpretation errors and reporting errors. 

- Standardize on a common COTS database application. Oracle database is 
most commonly used in PRACA and would be a good choice. 

- Implement standard user authentication across systems. 

- Extend the ADAM data warehouse to include relevant non-PRACA databases. 

- Decrease dependence on external data sources, find out why they exist, and 
incorporate or cross-index what is needed into the PRACA system. 

- Require that all SRQ&MA reports be generated using these databases to 
enforce the migration of all necessary data into the ‘ PRACA System. 

Simplify and standardize the user interface to allow ease of data access, cross- 
system navigation and data analysis with managed knowledge sharing. 

- Implement a standard GUI across all systems. Use a widely distributed and 
supported web browser as the foundation of this interface. 

- Implement transparency to isolate the user from database to database 

navigation. . . , 

- Implement a personalizable User Interface allowing customization of the 

interface to the needs of each user. 

- Provide collaborative capabilities to permit and encourage sharing and queries 


Oliu anaij^wo. odAjP A 

- Create data mining and reporting tools to support the advanced SRQ&MA 
analysts as well as the SSP management level overviews of the data. 

Utilize consistent and accessible secure network and system technologies to 
protect the data and the user access. 

- Develop a consistent security model for all data, networks, and systems 
associated with the PRACA System. 

- Identify and establish a security requirements document for the PRACA 


systems and their data. 

- Eliminate unnecessary data filters and network security bottlenecks. 

- Implement standard system authentication and encryption across systems. 

- Standardize on a common network architecture. 

- Transmit the data on a secure NASA-wide area network implementation. 
Leverage existing data mining tools and expertise to enhance the available 
trending, assessment, and analysis. 

- Automate repetitively generated reports and trend analyses. Identify ways o 


codify the labor-intensive procedures. 

- Increase the breadth, depth, accuracy, and speed of PRACA data analysis 
advanced automated and intelligent search techniques. 
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3 Develop a clear set of SRQ&MA trending and analysis requirements. Then develop 
requirements for the raw PRACA data to be collected to allow the SSP to make risk 

and safety assessments. , ^ f __ 

. Remove the sole dependence on human experts and corporate knowledge for 

problem assessment. 

• Determine the requirements for Program-wide SRQ&MA view of PRACA data. 


• Identify and fix the problems causing data interpretation and “scrubbing” for 

report generation. ° 

• Require that all Safety and Risk data reports be generated using this system to 
enforce the migration of all necessary data into the PRACA System. 

4. Train and inform personnel in all of the levels of the PRACA system on the 
processes, motivation and importance of the PRACA data and the system. Analyze 
the work processes and implement changes to accommodate the SSP PRACA vision. 

• Create end-to-end electronic collection, capture and management of problem 
reports to reduce PRACA data capture and entry errors. 

• Incorporate PRACA reporting interface use as part of a data collection quality 
improvement process. 

• Extend the work process assessment to include other PRACA sites, including 
Marshall Space Flight Center, Palmdale, and the Huntington Beach Problem 
Analysis Center, and expand the study of JSC and KSC processes to include 
observational as well as interview data. 

• Re-evaluate the strict hierarchy of problems, based on the tree structure of the 
Shuttle assembly. This hierarchy makes it difficult to document or describe 
problems that result from interactions between components in different 
assemblies or systems. 

• Institute training of technicians and engineers in Program-wide PRACA and what 
kinds of information are being requested and why. 

- Resolve local differences in how different organizations fill out Problem 
Report fields. 

- Resolve differences between organizations in how they categorize problems. 

• Determine why there is so much paper movement, and which of it could better be 
accomplished electronically. 

— Some of the work being done appears to be more easily and accurately done 
by a computer than by a human. 

- Evaluate the potential for electronic transfer of all documents and the ability 
to sign the forms on-line with a password protected electronic signature. 

• Determine if, as suggested, a measure of organizational accountability is “the 
number of problem reports filed.” 

— If true, this affects the report classification decisions. This would tend to 
create a work climate where reducing the number of Problem Reports filed, by 
tending to identify a nonconformance as a less significant category, has 
incentive. This would skew the data in the PRACA systems. 

We believe that an Agency-wide NASA/Industry team in conjunction with the SSP 
PRACA workforce can bring together the required expertise, knowledge, and advanced 
IT capabilities necessary to achieve NASA’s Information Management vision for 
PRACA. In so doing, PRACA will remain a critical and vital system, enabling a 
reduction in the risk and improvements in safety while supporting the Space Shuttle 
Program into the next decades. 
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1.0 Motivation and Approach 


1.1 Motivation for This Study 

The PRACA Enhancement Pilot Study was initiated in response to the Fall 1999 review 
by the Shuttle Independent Assessment Team (SIAT). The first part of the Pilot Study 
was an initial NASA Ames Research Center Problem Reporting and Corrective Action 
(PRACA) Technical Review performed in January 2000 at the request of the NASA 
Code M Enterprise. The initial Technical Review was followed by a more detailed four- 
month study of the Space Shuttle Program (SSP) PRACA system. 

The SSP PRACA systems and their supporting infrastructure are used to report 
discrepancies, non-conformances, problems, track corrective action, and extract data for 
trend analysis and reporting. It is an essential system for managing Shuttle safety and 
readiness for flight. The charter document that describes the requirements for the SSP 
PRACA system is NSTS 08126, Revision G [ref. 1]. 

Because of their importance, the PRACA systems have been the subject of several recent 
reviews aimed at improving the systems utility and improving the motivating 
requirements. Two of these reviews, the SIAT Report and die initial ARC PRAC 
Technical Review, are described in the following two sections. 

1.1.1 Shuttle Independent Assessment Team 
Report 

In September 1999, NASA chartered the SIAT to provide an independent review of the 
Space Shuttle sub-systems and maintenance practices. The SIAT published its report in 
March 2000 [ref. 2]. In the SIAT report, several findings and recommendations were 
raised specifically regarding the SSP problem reporting practices and systems that may 
adversely affect Shuttle safety. 


1.1. 1.1 SIAT Problem Reporting Findings 

1. The Problem Resolution and Corrective Action reporting system appears designed 
from the perspective of data to be kept (“bottom up”), not from the perspective o 
decisions to be made (“top down”). It does not provide high confidence that all 
potentially significant problems or trends are captured, processed, and visible to 
decision-makers. 

2. Effective utilization of the Problem Reporting and Tracking system requires 
specific expertise and experience to navigate and query reporting systems and 
databases. 
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3. Missing and inconsistent events, information, and criticality lead to a false sense 
of security. 

4. Tracking and trending tools generally lack sophistication and automation, and 
inhibits decision support. Extensive “hands-on” examination and analysis is 
needed to process data into meaningful information. 

5. Critical information may be lost and ignored, and problems may be repeated due 
to weaknesses in reporting requirements, and processing and reporting 
procedures. 

6. The fragmented structure of the Problem Resolution and Corrective Action 
system, built from legacy systems, minimizes its utility as a decision tool. 

1.1. 1.2 SIAT Problem Reporting Recommendations 

1. The SSP should revise the Problem Resolution and Corrective Action database to 
include integrated analysis capability and improved problem classification and 
coding. Also, improve system automation in data entry, trending, flag ing of 
problem recurrence, and identifying similar problems across systems and sub- 
systems. 

2. The root cause(s) for the decline in the number of problems being reported to the 
Problem Resolution and Corrective Action system should be determined, and 
corrective action should be taken if the decline is not legitimate. 

3. The root cause(s) for the missing problem reports from the Problem Resolution 
and Corrective Action system concerning Main Injector Liquid Oxygen Pin 
ejection, and for inconsistencies of the data contained within the existing problem 
reports should be determined. Appropriate corrective action necessary to prevent 
recurrence should be taken. 

4. A rigorous statistical analysis of the reliability of the problem reporting and 
tracking system should be performed. 

5. Standard repairs on CRIT1 components should be completely documented and 
entered in the Problem Resolution and Corrective Action system. 

6. Reporting requirements and processing and reporting procedures should be 
reviewed for ambiguities, conflicts, and omissions, and the audit or review of 
system implementation should be increased. 

7. The Problem Resolution and Corrective Action system should be revised using 
state-of-the-art database design and information management techniques. 

8. All critical databases (e.g., Waivers) need to be modernized, updated and made 
more user friendly. 

As a result of the SIAT report, a PRACA Evaluation Team (PET) was established at 
NASA Johnson Space Center (JSC) by Program Review Control Board Action 
S060341R5(3-1) [ref. 3]. Several tasks aimed at improving the PRACA requirements 
and compliance were initiated within each of the SSP’s Centers: Marshall Space Flight 
Center (MSFC), Kennedy Space Center (KSC), and JSC. Several activities were also 
initiated by the United Space Alliance (USA) to assess internal company PRACA 
requirements and compliance. 


13 


1 . 1.2 


ARC PRACA Technical Review 


The NASA ARC PRACA Technical Review was the initial phase of this Study and was a 
two-week effort at an initial qualitative assessment of the state of the PRACA system. A 
set of interviews with the SSP’s known PRACA stakeholders were held to determine the 
existing requirements, capabilities, and status of the PRACA system. Upon completion 
of the review (Jan 18, 2000) a presentation of results was made to the SSP office and the 
SIAT committee [ref. 7], identifying specific comments, and proposing potential areas of 
PRACA enhancement. 

The JSC PET has used the ARC Technical Review comments and observations as 
additional criteria to address in their evaluation. The observations attributed from the 

ARC review were: „ 

1 The Shuttle PRACA system is made up of several parts, which are currently an 

run independently. 

2. There are separate development activities for all PRACA systems (Program an 
Project). 

3. There is no Shuttle Program PRACA owner. 

4. The Project PRACA teams do not report to any Program Manager. 

5. The functional requirements (i.e.. what information must be tracked) for the 
PRACA systems come from two places. 

a. Only the SSP Office document NSTS 08126 Revision G defines the flight 
critical information to be tracked. 

b. The separate Center Projects and requirements exist at JSC, KSC, and 
MSFC that support the Shuttle Program. These are non-consistent 
requirements (i.e., they are not in conflict, but they are not necessarily 
complimentary). 

6 The integration of the various PRACA data from the Center Projects is integrated 
into two Program accessible systems; the old system is the Program Compliance 
Assurance and Status System (PCASS); the new system is called ADAM and is 
proposed as the basis for WebPCASS. 

7. There is no schedule or milestones for transition from PCASS to ADAM. 

8 There is no Shuttle Program owner or user of ADAM. 

9. Extensive knowledge of the PRACA systems and desired data is required for 
efficient operation and queries. 

10. Four particular areas could be assessed and improved; 

a. Front-end user interface for searching, displaying, and analyzing the 

PRACA data; __ A _ . . t 

b. Database and system infrastructure for storing/accessmg the PRACA data 

from the various Center systems; 

c. Problem reporting processes for capturing PRACA data; 

d. Requirements and management for Shuttle Program Office PRACA. 
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A new NASA Initiative, Design for Safety (DfS), was launched simultaneously with the 
release of the SIAT report. The PRACA Enhancement Pilot Study was started by NASA 

within the DfS Program to understand and detail the PRACA issues and 

recommendations raised in the SIAT report. The primary goal of this Pilot Study is to 
evaluate and quantify the technical issues with the current implementations of the SSP 
PRACA systems and recommend high value enhancements to address deficiencies and 
enable a future Safety and Risk prognostics capability. 

1.1.3 The Creation of the SSP PRACA System 


In 1987, after the Challenger accident and in response to the Rogers Commission 
recommendation to provide NASA Space Shuttle management and decision makers with 
readily available, timely, and accurate data, the PCASS was formed. The PCASS is 
defined in document “System Integrity Assurance Program,” NSTS 07700 Volume XI. 

The NSTS 07700 document section 3.4 states that all Shuttle nonconformances, including 
unexplained anomalies, shall be documented and transmitted to the design project 
elements for investigation and resolution in accordance with the requirements of NSTS 
08126. PRACA data and status shall reside in or be accessible via the PCASS as 
specified in NSTS 07700 vol XI, section 4.1.x. 

The NSTS 07700 goals were to impact Shuttle processing, safety, and readiness for flight 
by enabling continuous process improvements. The PCASS-hosted overall “PRACA 
System would allow users access to the current and historical data necessary to perform 
trend analysis and reporting to aid in the process planning and improvement. To provide 
the data necessary for this Program-wide “PRACA System,” currently a combination of 
paper records, on-line databases from separate PRACA systems, and corporate/expert 
knowledge and skilled personnel are required. 

The PCASS is being replaced by an updated web-based version called WebPCASS. It is 
being proposed as a straightforward re-hosting of the mainframe-based PCASS onto a 
Unix server with a browser interface. The NSTS 07700 goals of an interactive Shuttle 
data store for use in trending, safety and reliability analyses are not yet being realized. 

1-2 Assessment Approach 


The objective of this Study report is to document a quantitative assessment of the 
technical and operational status of the SSP PRACA systems and elements. This 
quantification is intended to enhance the initial ARC Review’s 1-month qualitative 
assessment (completed in January 2000). In addition to the “as-implemented” aspects, the 
team desired to understand the “as used” issues and challenges with the PRACA systems 
so that any recommendations, while technically feasible, can also be evaluated for their 
practicality and work environment utility. 



The approach taken by the Study team was to interview, understand, and assess. This 
approach required multiple site visits, telecons, and interviews with as many of the 
people involved in the PRACA system as possible (managers, users, customers, etc). 1 he 
team consistently noted the support and cooperation by the NASA and contractor staff 
throughout the SSP PRACA system. This was fairly unique in the team’s experience to 
see such cross-center cooperation and enthusiasm for progress towards a common goal. 
All of the team’s requests for information, documentation and time were professiona y 
addressed and met the team’s needs. A summary list of contacts, sites, and interviews is 

provided in the Appendix. 

Through the course of our Study, the team continually coordinated its activities with the 
JSC PET. Specifically we kept Ms. Linda Ham, supporting Mr. Ronald Dittemore; 
Manager of the Space Shuttle Program, appraised of our status, observations and 
findings. 

The quantitative technical assessment began in March 2000. The Team visited and 
interviewed PRACA systems owners and users at JSC, KSC, and MSFC. The purpose o 
these meetings was to understand how the SSP elements collect, manage, and use e 
problem reporting data. The team also interviewed multiple safety, reliability, quality 
and mission assurance users of the PRACA data to determine the desires and implicit 
requirements for the PRACA systems. As part of the interview process, the team 
collected available system documentation recommended by the contacts and thought to 

be of value to the study. 

Based upon the SIAT report findings, and to simplify the organization of the report, the 
team decided to group the PRACA system technologies into four primary technical areas. 

These four areas are: 

1. User Interface 

2. Database and data management 

3. Network and system architecture 

4. Work processes of Problem Report generation and use 

The focus of these areas is described in greater detail in the following sub-sections. 

1.2.1 User Interface Assessment 

Because of the SIAT emphasis placed upon increased access and visibility into the 
PRACA data (i.e., broader NASA and Shuttle user community access, access to greater 
detail and supporting data, need for cross PRACA data mining, increased user- 
friendliness, etc.) the user interface was chosen as a primary technology area of study. 

The user interface was studied from several user perspectives. 

• The PRACA system data manager and administrator. For this user, database 
administration, data security, entry, management, and control are the primary 
interface uses. 


• The SRQ&MA user. This user’s main interests are to produce knowledge and 
conclusions via data extraction, reduction, analysis and trending. For this user, the 
interface is a tool to simplify navigation of the databases to find the obvious and 
not-so-obvious relationships and to assist in the generation of meaningful reports. 

• The engineer/researcher interested in process and procedure improvement. This 
user is typically looking for hidden trends and data relationships, or is performing 
detective work. The user requires an intuitive way to navigate the data and follow 
links between databases to uncover otherwise “hidden” data trends. This user is 
typically the one to uncover or preclude the “escapes” and “diving catches.” This 
user requires an interface providing data mining and drill down capabilities with 
advanced analysis recording and sharing methods. 

• The high-level manager. This user is primarily interested in fast and eas\ ;cess 
to bottom-line conclusions, current and historical summary information and trend 
reports. For this user, the interface is used to navigate a report archive. 

The team interviewed the various user types and used and evaluated the various PRACA 
systems interfaces where possible. In the case of the JSC government-funded equipment 
system and the KSC group support system, the team did not achieve hands-on access, due 
to the team’s remote location exclusion from the firewall-protected LAN (security model) 
and the user interface being designed for local access only. 


1.2.2 Database and Data Management 
Assessment 


The quality of PRACA system data analysis and conclusions are dependent upon data 
integrity, quality, ease of access, and cross-system data query for full leverage of 
PRACA. Because of this, the Database design and implementation was chosen as a 
primary area of study. To perform the database assessment, the team considered many 
aspects, including: 

• Database application (relational, object oriented, web enabled, query languages 
supported). 

• Architecture or schema (tables, objects, multimedia capability, text treatment, 
entity-relations model, etc.) The team requested the database schema or design 
documentation, when available, and the naming conventions for the data fields. 

• Administration and data entry. The team determined the methodology for entering 
data into and retrieving data from the database, user management and access 
control, as well as the query methods and consistency across systems. 

No detailed assessment of the database software performance upon the computer systems 
was made at this time. 



1.2.3 


Network and System Architecture 
Assessment 


The highly networked NASA and aerospace community, increasing breadth of internet 
communication methods (text, audio, video), new network security models, and advances 
in system architectures and performance, compelled the team to select network and 
system architecture as the third primary area of study. The team considered many aspects 
including: 

• Server technology 

• Network security implementations (firewall, proxy, trusted host, etc.) 

• Center-to-center communication techniques 

• Technology maturity levels relative to state-of-the-art. 


1.2.4 PRACA Work Process Assessment 

The final primary area of study selected by the team was the PRACA work process. The 
team chose this area in an effort to ensure that its IT recommendations are consistent with 
the practical aspects of the real work environment. Work process assessment in this Study 
was limited to two PRACA sites: on-site tracking and interview of the KSC PRACA data 
collection, and JSC Orbiter problem reporting closure processes. The team believes this 
preliminary assessment of two PRACA centers demonstrates the utility of a work process 
study and that a more detailed assessment can be performed in the future at additional 
PRACA sites. 

To perform the work process assessment, the team began by performing a series of 
interviews at JSC and KSC to obtain an initial analysis of relevant work process and 
work flow issues. The PRACA work process assessment focused on the KSC PRACA 
data collection, and JSC Orbiter problem reporting closure processes. Follow-on 
assessments should perform observations of the actual process of PRACA reports being 
initiated, dispositioned, filed, transferred, and used. 

1. 2.4.1 Work Process Assessment at JSC 

Interviews at JSC were focused on two levels. First the team investigated the databases 
to determine how they are used. Second, the team investigated how the organizations 
that support and use these databases interact. This includes the relation between the 
Space Shuttle Vehicle Engineering Office (which has the final say on Shuttle flight 
constraints) and the Orbiter PRACA database (whose database is owned by USA and 
whose work process and front end are owned by Boeing). It also includes the JSC 
Government Flight Equipment (GFE) database. In addition, we interviewed members of 
the Shuttle Safety Assurance and Mission Assurance office. In these interviews, we 
attempted to determine the flow of information through the PRACA system, and the 
perspective of the users in each area. 


1 .2.4.2 Work Process Assessment at KSC 


Interviews at KSC were designed to focus on the process by which a problem is 
discovered and a PRACA problem report is filed. The team attempted to determine the 
exact work of filling out the form for a problem report from the initial discovery of a 
nonconformance to final closing out of the report in the database. 


2.0 State of the SSP PRACA System 

This section details the state of the SSP PRACA systems. This information was collected 
from the interviews with various PRACA system owners and the Project/Element level 
and analysis of available PRACA documentation. 

2.1 PRACA System Overview 


The Space Shuttle PRACA systems and their supporting infrastructure are used to report 
discrepancies, non-conformances, problems, track corrective action, and extract data for 
trend analysis and reporting. It is an essential system for managing Shuttle safety and 
readiness for flight. The charter document that describes the requirements for the SSP 
PRACA system is NSTS 08126, Revision G [ref 1], 

2.1.1 PRACA Requirements from NSTS 08126 Rev. 
G 

The NSTS 08126 Revision G document was signed on February 2, 1996. That document 
provides the minimum requirements and responsibilities applicable to all SSP PRACA 
systems, as required by NHB 5300.4(lD-2), Safety, Reliability, Maintainability and 
Quality Provisions for the SSP. The objectives of NSTS 08126 Revision G are to: 

a. Establish uniform standards to ensure safety, reliability, and quality of SSP. 

b. Establish the requirements/procedures to assure problems are dispositioned prior to 
flight. 

c. Ensure appropriate corrective action is taken in a timely and cost effective system. 

d. Provide the problem data necessary to support engineering analyses and logistics 
management. 

The NSTS 08126 Revision G does not address the issues or requirements for extracting 
the data necessary for trend analysis and reporting. While it does provide the data for 
engineering analyses and logistics, this has not proved sufficient to the SRQ&MA 
analysts and has necessitated having adjunct data sources and databases to perform risk 
and reliability trend analyses. 

NSTS 08126 Revision G has established some uniform standards for ensuring safety, 
reliability, and quality of SSP, but has not required or documented the critical 
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information and data standards. These include data management standards, such as field 
namin'* and database schema, that would enable the development of common SSP- wide 
PRACA data systems. Such a system is discussed in the recommendations, and is what 
the current implementation of the SSP PRACA lacks. The use and design of the existing 
PRACA systems, described in the following sections, points to the innate desire of the 
SSP for a unified access to PRACA data. 


2.1.2 Implementation of SSP PRACA 


The current SSP “PRACA System” is a collection of computer hardware, software, 
networks, and databases, as well as extensive paper files distributed across several 
NASA, USA and other contract support sites. These separate PRACA systems are 
managed by various teams of individuals whose job is to maintain the systems, keep t e 
databases current, and assist in the extraction of data for trending and reporting. One o 
the key components contributing to the success of the current PRACA systems is this 
support staff that forms the extensive corporate and institutional knowledge necessary to 
analyze, reduce, produce conclusions, and report results from the PRACA databases. 
Without these highly trained and expert staff, the utility of the PRACA data is reduce 

significantly. 

The data management capability behind the SSP PRACA System is a conglomeration of 
Proiect-level database systems to meet today’s Program-level requirements. Many of the 
component systems are designed as workflow management tools with strict requirements 
for streamlining accurate and timely problem resolution. Other systems are focused upon 
capturing the problem report data at a high fidelity useful in safety analysis and data 
minin'* applications. It is these two dissimilar requirements (workflow efficiency vs. 
extensive problem documentation) that are at odds in the current SSP PRACA systems. 


For the purposes of SSP use, the PRACA system is viewed as a collection of domain- 
expert managed systems. At the time of this Study most of the data from all but one of 
the Project PRACA systems could be queried through a centralized data warehouse 
(ADAM) for summary and condensed report viewing. However, as noted by the SIA , 
ADAM cannot be data mined or navigated by the non-expert user. As a result, the SSP 
uses the PRACA systems principally by contacting the appropriate responsible domain 
experts and requesting that a report or data trending exercise be produced for the Program 
office. Then, a team of domain experts at JSC, KSC, MSFC and other support sites 
accesses the PRACA component databases, other experts, additional off-line databases, 
and paper records to produce a report for SSP. These reports are usually placed in the 
PRACA systems and become part of the on-line record. It is important to note that reports 
of any sophistication (i.e., data mining, detailed cross PRACA correlations etc.) cannot 
be generated in real time using the on-line PRACA systems alone. 

The primary component systems and the information flow from the SSP expert centers 
are shown in the figure below: 
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Figure 1 - SSP PRACA Data Flow [S. Ahrens] 


2m2 State of the System (Spring 2000) 

In this Study we have identified four operational element PRACA systems and a fifth 
nascent Program-wide system. They are: 

• JSC’s Orbiter PRACA Data Support System (PDSS) 

• JSC’ s Government-Furnished Equipment PRACA system 

• MSFC’s Problem Assessment System 

• KSC’s Shuttle Problem Data Management System 

• USA’s Advanced Data Acquisition and Management system for PRACA Data 

2.2.1 JSC Orbiter (PDSS) 

The JSC Orbiter PRACA Data Support System (PDSS) tracks all reportable problems 
that occur on hardware for which JSC has design responsibility. PDSS provides the 
primary source of data used by the Program SR&QA Trending and Analysis group (JSC 
Code NC and SAIC). Using the PDSS the JSC Problem Action Center team can access 
more than 55,000 Failure Records which includes the over 20-year history of Incoming 






Messages (IMs), Suspect Problem Reports (SPRs), Corrective Action Records (CARs), 
and Non-Flight Constraints (NFCs). The PDSS summarizes failure history, performs 
trending analysis and provides management with visibility through sorted reports. 

The PDSS PRACA requirements are documented in “Procedures for Orbiter Problem 
Reporting and Corrective Action (PRACA)” USA document PAC-2718283 (Revision B). 
The PAC-2718283 document applies to all elements and sites involved in the 
manufacture, assembly, handling, use, testing, or repair of any Orbiter component. Shop 
Replaceable Unit (SRU), or Line Replaceable Unit (LRU). It is specifically applicable to 
the USA’s prime contractor and subcontractors. It also includes Ground Support 
Equipment (GSE) for which USA has design responsibility. 

As with all PRACA systems, overall requirements for establishing a closed loop 
problem-reporting system are documented in NHB 5300.4(lD-2) Safety, Reliability, 
Maintainability and Quality Assurance Provisions for the Space Shuttle Program. The 
Level II requirements for implementing this system are defined in NSTS 08126, SSP 
PRACA System Requirements. 


A summary of the PDSS specifications and key information are shown in the table below: 


System Name: 

Orbiter PRACA, 

PRACA Data Support System (PDSS) 

Point of 
Contact: 

NASA: David Brown 
Contract: Suzanne Little (USA) 

Location: 

Primary system: NASA JSC 

Support sites: Palmdale, Downey, KSC, and NSLD, MSFC 

Operational 

Since: 

More than 20 years. PDSS contains data from as early as 1974. The 
GFE data was originally part of the PDSS database. GFE was 
established as a separate database in the 1997-1998 time frame. 

In 1998 the database was upgraded to Oracle. 

Purpose: 

Tracks all reportable problems that occur on hardware for which JSC 
has design responsibility 

Platform: 

IBM PC Compatible 

O/S: 

WinNT 4 

Database: 

Oracle 7 (upgrade from DB2) 

Application 

Programs: 

Client App: Gupta SQLWindows 

DB Management: Integrated Database Application (IDA) (custom) 
Reporting: Trendtool (custom software) 

Security 

Model: 

Primary: Firewall Protected Intranet. (Mirrored at USA and Boeing for 
intranet access) 

Passwords/UDD: Read: None, Write: User and Password required 

Requirements 

Document: 

NHB 5300.4 (ID-2), 

NSTS 08126, 

P AC-27 1 8283 (updated to Revision B on May 22, 2000) 

Web Info: 

HttpV/www.houston. ssd-bna.boeine.com/d331/pac/mainpdss.htm 
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Table 1 - JSC Orbiter System Information 


2.2.1. 1 Architecture 

The PDSS has followed the Space Shuttle development with the earliest data in PDSS 
originating in 1974. The first version of what was to become the SSP PDSS was created 
by Rockwell (though earlier versions may have existed for the Apollo Program). With 
contract changes, the system moved from Rockwell to Boeing ownership, then to its 
current owner, Boeing-USA. The PDSS originally contained the GFE database 
information. Most GFE data were removed from PDSS three years ago when the decision 
was made to separate task management of NASA-owned GFE from Program-owned (and 
thus USA-managed) GFE. The GFE owned by the Orbiter Project office remains in 
PDSS while NASA-owned GFE is managed in the GFE database (discussed in the next 
section). 


2.2.1. 1.1 System 


The PDSS database is housed on a single IBM-compatible computer system (the Host) 
running the Windows NT operating system located at the NASA JSC Boeing-USA 
facility. The computer system runs the Oracle 7 relational database application and can 
support multiple databases and several simultaneous users. This Host system is connected 
to the NASA JSC Firewall Protected Local Area Networks (LAN). There are no 
passwords or User ID required for read access but valid User and Password are required 
for write and administration access. For increased user community access while 
maintaining the network security model, a second database system is mirrored at the USA 
facility. 

The client machines access the PDSS database by running a Gupta Corporation 
SQLWindows-based application developed by the Boeing-USA PDSS support group. 

The data can be queried and reports generated from either the Host or the authorized 
client computers. Database management requires user and password authentication and 
can be performed on the Host or trusted client systems within the firewall-secured LAN. 

2.2.1. 1.2 Network 

PDSS PRACA is implemented on the JSC internal web, which is protected by firewall 
security. The host allows specific computer connections via a client access table managed 
in the Host computer. Access at firewall-separated LANs is currently managed by 
mirroring the system across the firewall. The system and user access tables maintained in 
the Main Host computer provide access control. There is no encryption of the data as it is 
sent across the network. User management is independent of the other SSP PRACA 
systems and is managed locally by the PDSS group. 

2.2.1 .1 .3 Database 


23 



The Orbiter database was originally implemented in DB2 [ref 11]. The PDSS uti lzes 
several process models and a single database. The Orbiter PRACA process models 
include process flows for the initial CCAR/TCAR decision and the subsequent 
processing of these problems. The database uses twelve primary tables with several 
additional tables to support applications processing and codes. There is one entry in t e 
common table for each problem record. There are repeating entries in five other tables for 
each problem record. There are six tables that contain narrative text information. The 
field JSC REPORT_# is the unique identifier for each problem record and exists in eac 

table. 


The system was transition to the Oracle 7 relational database in 1998 dunngthe 
extraction of NASA-owned GFE data (and simultaneous creation of the GEE database). 
The basic twelve relational table design has been preserved while adding additional 
support tables. 


System management is currently performed using custom software called the Integrated 
Database Application (IDA). The IDA controls data input to PDSS through required 
input fields and checks the inputs against allowable tables. The IDA ensures data entry is 
complete and accurate. Access is limited to the Problem Action Center by firewal , 
system, user and password authentication. Approximately 5-6 people currently have such 
access. The current size of the database is approximately 5 GB. 


2.2.1. 2 Data Collection 

The data collection process is described in PAC-2718283 (rev B) “Orbiter Problem 
Reporting and Corrective Action”. The data collection process flow is illustrated in the 

figure 2 below: 
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Figure 2 -NASA JSC Orbiter Nonconformance/PRACA Process Generic Flow 


Database entry is limited to the Boeing Product Assurance Problem Action Center. The 
Problem Action Center was chartered in compliance with the provisions of NASA NHB 
5300.4 (ID-2) and the implementation requirements of United Space Alliance (USA), “to 
facilitate an ongoing centralized system for the timely reporting of significant Orbiter 
Hardware nonconformances and to administer the resultant problem reports until 
adequate Boeing disposition can be accomplished in support of the Space Shuttle 
Operational Program.” 

The Problems being reported into the PRACA system through the Problem Action Center 
include significant pre-ATP and ATP failures occurring at suppliers and Boeing 
manufacturing facilities: ~ 

• During supplier and/or Boeing certification/ qualification testing; 

• At Boeing facilities and the NASA test and Operational centers during ground 
test, In-Flight, turnaround and overhaul; and 

• During repair operations or shipping and receiving to or from repair depots. 

The failed hardware includes flight hardware, like-flight hardware, spares and safety- 
critical ground support equipment. 

2.2.1. 3 Interface 

The PDSS summarizes failure history, performs trending analysis and provides 
management with visibility through sorted reports. The system can be queried for data 
ranging from general trend studies to specific failure history searches for individual part 
names, part numbers, serial numbers and selected causes of failure. The data in PDSS are 











accessible via custom client applications written using Gupta SQLWindows. These 
applications are provided by the PDSS support group and run on IBM PC compatible 
computers with the Windows OS environment. The PDSS support group is currently 
looking into implementation of a fully web-enabled interface. 

The PDSS supports a custom reporting and trending application called Trendtool. 
Trendtool is a windows-based menu-driven, on-line, interactive application that provi es 
access to trending capabilities for engineering analysis. Its capabilities include on-line 
viewing, printing, interactive plotting and electronic file generation. 


In addition, the JSC Problem Action Center (PAC) generates weekly reports reflecting 
PRACA status. Prior to each launch, management is furnished with the L-10 through L- 
day report daily, identifying all open issues pertaining to the launch. The PAC maintains 
a distribution list for these reports. The PAC established a PRACA web page to provide 
open access to PRACA documents, reports, and procedures. The PRACA web page is 
maintained by the PAC-Houston Operation and may be viewed at: 

hrrp://www Hniiston.SSD.BNA.B o eing.com/D331/PAC/MainPDSS.htm 


2.2.2 JSC Government-Furnished Equipment 
(GFE) PRACA system 

The Government-Furnished Equipment (GFE) PRACA database is another essential 
component system for the SSP’s PRACA data. The GFE system tracks all reportable 
anomalies and non-conformances detected on JSC GFE flight hardware, and equipment 
that is representative of flight hardware. The primary contractor for support of this system 

is SAIC. 

The GFE PRACA requirements are documented in JSC 28035 (currently under revision). 
As with all PRACA systems, overall requirements for establishing a closed loop 
problem-reporting system are documented in NHB 5300.4(1-D2) Safety, Reliability 
Maintainability and Quality Assurance Provisions for the Space Shutt ^ P ™f am R e 

Level H requirements for implementing this system are defined in NSTS 08126 Rev. O. 


Key system information is presented below: 


System Name: 

Government Furnished Equipment (GFE) 

Point of 
Contact: 

NASA: David Dyer 

Contract: Scott Ferguson, (SAIC) 

Location: 

Primary system: NASA JSC 

Support Sites: none — 

Operational 

Since: 

GFE contains data from as early as 1974. The GFE data was originally 
part of the Orbiter PRACA database. GFE was established as a 
separate database in the 1997-1998 time frame. 

Purpose: 

Tracks all reportable anomalies and non-conformances detected on 
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JSC GFE flight hardware, and equipment that is representative of flight 
hardware 

Platform: 

IBM PC Compatible 

O/S: 

WinNT 4 

Database: 

Microsoft Access 97 

Application 

Programs: 

Client App: Runtime version of MS Access 97 
DB Management: MS Access 97 (Main Host only) 
Reporting: MS Access 97 (Client and Main Host) 

Security 

Model: 

Primary: Firewall Protected Intranet. Machine level access table 
maintained on Main Host 
Passwords/UID: None 

Requirements 

Document: 

NHE 5300.4 (ID-2), 

NSTS 08126, 

JSC 28035 (currently under revision) 

Web Info: 

None Available 

Other: 

1 he GFE database provides support for both the SSP and ISS 
programs. Data for both programs reside in the same database. 
Program data are distinguished by the “ProgramAssignment” field 


Table 2 - JSC GFE System Information 


2.2.2.1 Architecture 


The GFE database was originally part of the Orbiter PRACA system (as described in the 
section above on PDSS). Like PDSS, the GFE data has followed the Space Shuttle 
development with the earliest data in GFE originating in 1974. The GFE data were 
separated out of Orbiter PRACA during the PDSS Oracle upgrade in the 1997-98 time 
frame. This upgrade left two separate data systems (GFE and PDSS) resulting in separate 
data management responsibilities. The GFE database contains information about NASA- 
owned GFE, while the SSP-owned GFE is tracked in PDSS. 

In addition, the GFE database design supports both the SSP and ISS programs. Though 
the data from both programs occupy the same database tables, the database architecture 
allows for managing and retrieving each program’s data separately and presents no 
apparent operational problems. 

2.2.2. 1 .1 System and Network 

The GFE database is housed on a single IBM-compatible computer system (the Host) 
nr nng the Windows NT operating system. This host runs the Microsoft Access 97 
re. :onal database application and can support multiple databases. The host is connected 
to he NASA JSC firewall-protected LAN and allows connections via a client access 
table. The clients access the GFE database via a run-time version of MS Access 97. The 
data can be queried and reports generated from either the host or the authorized client 
computers. Database management is performed on the host system directly. 
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2.2.2.1.2 Database 


The GFE database is implemented in the Microsoft Access 97 database application. T e 
database architecture has 12 primary tables (Common, PARTS, REC_CNTL, 

REL DOCS, REV_SETS, SRA_SETS, Tbl_DeferralRationale, TbLDeferredStatus, 
Tbl_DRNumber, Tbl_FME AN umbers, Tbl_Remarks, Tbl_TPS). The 12 tables are legacy 
architecture from the original DB2 design of Orbiter PRACA. All tables share the 
common field “rept_no”. Additional Access database tables contain codes and support 

information. 

The populated GFE database size is currently ~ 100MB. There is no indication that GFE 
will outgrow available disk capacity. 

2.2.2. 2 Data Collection 


The data collection process for GFE is described in the “JSC GFE PRACA Process 
document authored by SAIC’s Scott Ferguson. When a nonconformance is determined to 
be PRACA reportable, the appropriate personnel initiate a PRACA report with a unique 
control number. The PRACA report is delivered to the JSC GFE PRACA Center (JGPC). 
The JGPC is responsible for distribution of the PRACA reports to the appropriate 
personnel, filing the appropriate forms (e.g., JF 2174E, JF 2174G, JF2174H contractor 
reports, etc) and entering/updating the information into the JSC GFE PRACA database. 

The process is shown in the flowchart presented below: 
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2. 2.2.3 Interface 


The trusted client machines access the GFE database by executing a run-time version of 
MS Access. The reporting and data query interface uses the MS Access GUI 
environment. The data can be queried and reports generated from either the Host or the 
authorized client computers. Database management is performed on the Host computer 
via MS Access directly. 


2.2.3 MSFC Problem Assessment System 

The MSFC Problem Assessment System is operated by Hernandez Engineering 
Incorporated (HEI) and reports to the MSFC Safety and Mission Assurance (S&MA) 
Problem Assessment Center. 

The MSFC PRACA requirements are documented in QS10-R-005 Revision B and the 
MSFC Problem Assessment Operations Plan, and the MSFC Problem Assessment Center 
Operating Instructions. The overall requirements for establishing a closed loop problem- 
reporting system are documented in NHB 5300.4(1D2) Safety, Reliability, 
Maintainability and Quality Assurance Provisions for the Space Shuttle Program. The 
Level II requirements for implementing this system are defined in NSTS 08126. The 
MSFC PAS also is responsive to the ISS 30223 document - Problem Reporting and 
Corrective Action System Requirements for the International Space Station (http.//iss- 

www.jsc.nasa.gov/cgi-bin/pals/docchk?52829). 


Key Information about the MSFC PRACA system is presented in the table below: 


System Name: 

Problem Assessment System (PAS) a.k.a. UPRACA 

Point of 
Contact: 

NASA: Alex Adams, QS-20 and Don Whirley, QS-10 
Contract: John W. McPherson (HEI) 

Location: 

Primary system: NASA MSFC 

Support sites: MSFC, JSC, KSC 

Operational 

Since: 

The MSFC Problem Assessment Center began in 1978 

Purpose: 

Tracks all 08126 PRACA variables reported from SSME, ET, RSRM, 
and SRB 

Platform: 

Sun Server 300 MHz, 1 Gbyte of RAM, 
33.6 Gigabyte HD space available 

O/S: 

Solaris Version 2.6 . 

Database: 

Oracle v 7.3.0.2 

Application 

Programs: 

Client App: written in C v 4.2 (custom SW) 
DB interface: embedded SQL in C program 
Reporting: HTML generation by C program 

Security 

Model: 

Primary: User authentication and NASA IP restricted web-site 
Passwords/UID: required for UPRACA data access 
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Requirements 

Document: 


Web Info: 


Other: 


NSTS 08126 
QS-10-R-005 rev B 
NHB 5300.4 (ID-2) 

NAS8-40364, DR 9 

ISS 30223 

MSFC Problem Assessment Center Page 
http://msfcsma3.msfc.nasa.gOv/tech/pac/s mapac.htmi 
MSFC UNIX PRACA Data System 

http://upraca.msfc.nasa.g ov:8018/praca/review/ni]hlic/html/inriftx htm 
Tracks SSP, Station and several other MSFC projects 


Table 3 - MSFC PRACA System Information 

2.2.3.1 Architecture 

The MSFC processing of Shuttle hardware prime contractor PRACA reportable problems 
is performed by a combination of civil service and contractor personnel within various 
organizations. This process flow is applicable to MSFC processing of External Tank 
(ET), Reusable Solid Rocket Motor (RSRM), Solid Rocket Motor, (SRM), and Space 
Shuttle Main Engine Problems. 

The element hardware prime contractor is responsible for problem reporting, statusing, 
and correction. These requirements are defined in the contractual documents between 
MSFC and the contractors. The MSFC Problem Assessment Center (PAC) runs the 
Problem Assessment System (PAS) as part of the Safety and Mission Assurance (S&MA) 
support services contract. The PAC does the following: 

• Receives the problem reports in the prime contractor format; 

• Reviews and validates the data for accuracy, clarity, consistency, and completeness, 
seeking additional data as required; 

• Translates the prime contractor data into the standard data field of the MSFC PRACA 
system (UPRACA) and the NSTS 08126 defined data fields; 

• Enters (re-keying and/or reformatting) the data into the PRACA database; 

• Facilitates the dissemination of the problem reports to the appropriate MSFC offices; 

• Monitors the compliance of the prime contractor against SSP and MSFC 
requirements; 

• Provides data, analysis, and evaluations in support of reviews. 

2m2m3m 1 ml SyStGffl 


The MSFC PRACA database is housed on a single Sun computer system (the Host) 
running the Solaris 2.6 operating system. Since this operating system is a Unix variant 
the PRACA database system is known as UnixPRACA or UPRACA. This system runs 
the Oracle 7 relational database application to maintain and manage the PRACA data. 
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This Host system is connected to a NASA MSFC firewall-protected LAN. A web server 
is used to provide client access to the UPRACA data system. The client machines access 
the UPRACA database by running a commercial browser on the client system. The data 
can be queried and reports generated from either the Host or the authorized client 
computers. Database management is performed on the Host computer directly. 

2.2.3. 1.2 Network 

MSFC’s PAS is implemented on the MSFC internal LAN, which is protected by firewall 
security There is a unique User ID and passwords pair required to access the system. The 
Webserver’s authentication of the user is used to manage access control. To gain access to 
the UPRACA data, a user must fill out a “PRACA System Access Request form at the 
online website at http://unraca.ms fr nasa . gO v: 8018 /p r aca/review/public/html/mdex.ht m- 
The PRACA System Administrator will approve the request and email a unique User ID 
and password to the user. Database management is performed on the host Sun computer 

directly. 


2.2.3.1.3 Database 

The MSFC Problem Assessment Systems UPRACA is based upon the Oracle 7 database. 
It has approximately 18,000 problem reports and in excess of 900 000 records T ^ h ' s 
sixteen table database is approximately 230 megabytes in size with the whole UPRACA 
database able to hold up to 5 gigabytes of data. All tables share the common field 
“MSFC_REPORT_#”. 

2.2.3. 2 Data Collection 

The MSFC PAC obtains the PRACA-reportable information from the four MSFC prime 
contractors’ electronic data systems. These prime contractors each cover a separate 
component of the launch system. The components are Space Shuttle Main Engine 
(SSME) External Tank (ET), the Reusable Solid Rocket Motor (RSRM), and the Solid 
Rocket Booster (SRB). The transfer of PRACA-reportable information from the 
contractors is achieved in one of several ways: email, fax or electronic transfer. 

The SSME PRACA data comes from Boeing North America/Rocketdyne Division in 
Canoga Park, California. An automated PRACA data file generation and electronic 
submission to MSFC occurs at night when appropriate. Rocktedyne’s system is the 
Problem Reporting and Management System on an IBM 390 running an IMS database. 

The ET data comes from USA (formerly Lockheed Martin) located at Michoud, 
Louisiana. A manual process starts the PRACA data file generation and electronic 
submission to MSFC PAS as required. MSFC manually invokes data loadupon 

notification of file transfer. The USA Corrective Action Process System (CAPS) 

PRACA report document is also faxed to MSFC for inclusion into the hardcopy file kep 
for each report. The NSTS 08126 required fields are added to the CAPS data and m the 

transfer file format. 
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The RSRM data comes from Thiokol located in Brigham City, Utah. They deliver a PAS- 
item PRACA document to MSFC via fax and e-mail of Microsoft Word documents. Not 
all of Thiokol s discrepancy reports are elevated to a PRACA-reportable item and very 
few NSTS 08126 data fields are included. 

The SRB data comes from USA (formerly USBI) at KSC. USA delivers PAS-item 
PRACA documents to MSFC via fax and e-mail of Microsoft Word documents. Not all 
documents in the USA Nonconformance Information System are elevated to a PRACA- 
reportable item and very few NSTS 08126 data fields are included. 

The data collection process flow always has the MSFC PAC loading electronically or 
keying a PRACA problem report into MSFC PRACA data syster with standard NSTS 

08126 Revision G codes and formats added. The full MSFC process flow is shown in the 
figure 4 below. 


Incident occurs 
involving flight or 
flight-like hardware 


Incident documented on 
detecting-organization 
form 


Problem responsibilities Prime hardware contractor 
passed to prime hardware documents problem on their 
contractor - 1 st level problem form 



Contractor dispositions 
issue separate from PAC 
reporting and review 
processes 


As it PAC 
reportable? 


Hardware contractor 
reliability group reviews 
problem for PAC 
reportability 
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Contractor develops 
analysis plan, 
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needed) interim flight 
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Interim 
Closure 
needed ? 
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Figure 4 - MSFC PRACA Data Collection Process Flow 

2. 2. 3.3 Interface 


The MSFC PAS provides an interface to the UPRACA system and PRACA data via a 
web-based access using any of the commercial browsers as the client front-end. The user 
interface was developed for the S&MA office and expert users. It is not developed as a 
general purpose or managerial support interface to PRACA data. The web-based online 
application is written in C that generates the HTML used for screen displays. Embedded 
SQL is used in the C program for the database calls. 


2.2.4 KSC Ground Operations PRACA System 

The KSC Ground Operations Support is operated by the United Space Alliance (USA) 
under the SFOC (Space Flight Operation Contract) to the SSP. The SSP PRACA 
requirements are documented in QA-OOl md QA-002 [ref 22 and 23], These documents 
define methods and responsibilities for documenting, dispositioning, and obtaining 
corrective action for problems encountered regarding hardware and software for which 
the KSC Space Flight Operation Contract (SFOC) operated by USA has responsibility. 
As with all PRACA systems, overall requirements for establishing a closed loop 
problem-reporting system are documented in NHB 5300.4(1D2) Safety, Reliability, 
Maintainability and Quality Assurance Provisions for the Space Shuttle Program. The 

Level II requirements for implementing this system are defined in NSTS 08126 Revision 
G. 


Key Information about the KSC PRACA System is presented in the table below: 


System Name: 
Point of 

Contact: 

Location: 

Operational 

Since: 

Purpose: 

Platform: 

O/S: 

Database: 

Application 

Programs: 


Securit\ 


Shuttle Processing Data Management System (SPDM S) PRACA 
NASA: Ruth Harrison, Randy Segert 
Contract : Daniel Mondshein, USA 
Primary system: KSC 

Support Sites: JSC and Palmdale via web interface only 

The PRACA data available dates from January of 1978 

The KSC SPDMS PRACA database is the repository for KSC ground 
operations PRACA data 

IBM 9000 with a Compaq computer ru nning Window as console 
VM 

SQL/DS 

Client App: Command line interface to database via terminal 
application 

Database Management: via command line and SQL DS tools 
Reporting: via command line or ADAM web-based query interface 
Primary: Network connectivity to the SPDMS system 




Model: 

Passwords/UID: Required for the SPDMS command line client and the 
ADAM web-based query screen — 

Requirements 

Document: 

NHB 5300.4 (ID-2), 

NSTS 08126, 

QA-001, QA-002, QA-019 - — 

Web Info: 

http7/usal unitedspacealliance.com/hQ/warehouse/ssp /oortecnnicdi/Ks 

c%5Fpr.htm 

h ttp://usapo 1 .ksc .nasa. eo v/ apps/usago/orgs/sreOQl/ 


and 

http V/www .usano.ksc .nasa. gov/APPS/usano/orgs/6 Li 


2x/supoortabilitv/index.cfm 

Other: 

Manages MRs, PR/DR/IPRs. Used for engineering disposition and 
workflow management. 


Table 4 - KSC PRACA System Information 


2.2.4. 1 Architecture 

The KSC Ground Operations Shuttle Processing Data Management System (SPDMS) 
houses another component of the SSP’s Problem PRACA System. The KSC SPDMS 
PRACA database is the repository for PRACA data. The database also contains Shutt e 
Payloads IPR's initiated during integrated operations. The current data set contains the 
non-archived data presently residing on the SPDMS computer from approximately 1 -Jan- 
1978 through the last 24 hours. 

2.2.4. 1m 1 System 

The KSC PRACA is part of the SPDMS system and is managed by USA Ground 
Operations. The SPDMS database is housed on a single IBM 9000 computer system (the 
Host) running the VM operating system. This system is controlled through a console 
application on a Compaq PC compatible computer running the Windows NT operating 
system. The database is implemented in SQL/DS that supports multiple databases for 
much of the ground operations activities. 

This Windows NT system is connected to the NASA KSC LAN and allows connections 
via terminals and remote connections. The terminals access the database by running a 
command-line interface to generate reports. The data can be queried and reports 
generated from either the terminals or the authorized remote connections using the 
command-line interface. Database management is performed on the LAN-connected 

terminals. 


2.2.4. 1. 2 Network 
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KSC PRACA on SPDMS is implemented on the KSC internal LAN, which is protected 
by a firewall. The IBM VM mainframe does not natively support IP (internet protocol) 
connections and is otherwise accessed via a Compaq computer running mainframe 
connectivity software. There are passwords and User ID required to access this front end 
Host. Database management is performed on the Host computer directly. 

2.2.4. f. 3 Database 

A relational database SQL/DS is used on the SPDMS to manage the PRACA data at 
KSC. The SPDMS holds at least nine separate databases for various group operations 
functions. There are 35 primary and support tables. Common fields are 
“Reference_Report_Number” and “USER JD”. 


2. 2.4.2 Data Collection 

The data collection process flow is described fn the chart below. 
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2. 2.4.3 Interface 

Access by analysts and the KSC PRACA team is via a command-line interface to the 
mainframe. There is a variant of the ADAM (see section 2.2.5) custom user interface to 
access a copy of the SPDMS mainframe system PRACA data. 

The SR&OA groups at NASA KSC have done reporting and trending in the past with 
varying degrees of user community acknowledgement and support. Reports can be found 
at httpV/usagal .ksc.nasa.g ov/apps/usago/orgs/sreQQl/ and 

httn //www.usa n n ksc.nasa.gov / APPS/usano/orPs/61-2x/supportabihty/index.c fm- 

The reports are generated by a group of data experts in consultation with the inspectors 
and engineers associated with the data acquisition and problem identifications. 

'38 



















2 . 2.5 


ADAM Data Warehouse 


The Program Compliance Assurance and Status System (PCASS) is defined by the 
System Integrity Assurance Program, NSTS 07700 Volume XI. It is currently in 
transition from a mainframe to distributed web-based architecture. This mainframe 
design limits the application of newer technology needed for enhancement of capabilities. 
The Problem Reporting interface in PCASS is provided by the Integrated Problem 
Assessment System (IPAS). The IPAS component, and most of the other PCASS 
systems, are in the process of being replaced with a web-based data warehouse 
architecture ADAM (Ad vanced Data Acquisition and Management). 

ADAM is a centralized data warehouse system that support PCASS concepts and the 
proposed WebPCASS project to provide access to data from numerous sources, with the 
goal to accurately analyze and assess the status of Shuttle pre-flight, flight post-flight 
activ s. The ADAM platform is built and maintained by the United Sp; Alliance 
(USA under the integrated Space Flight Operations Contract (SFOC) wit ne target to 
fulfill the PCASS requirements for an authoritative source for searching and reporting of 
NSTS 07700 defined data. & 

ADAM is being developed to apply new technologies (such as web interface and data 
warehousing) to fulfill expanding user requirements. The ADAM Data Warehouse’s 
stated goal is to: provide the SSP users the ability to access historical reliability 
performance; allow for the identification of patterns of deficiencies; allow development 
of statistical data to support continuous improvement of the fleet; and provide 
engineering data for determination of remedial and/or corrective actions (design 
improvements). USA desires to achieve this by assembling PRACA and other SSP 
quality and safety data into the ADAM data warehouse. ADAM is planned to provide the 
hardware and software infrastructure that enables the integration of multiple operational 
databases into a single database view designed specifically for reporting and analytical 

processing (refer to Figure 7). This is not yet implemented in the current instance of 
ADAM. 


Key Information about the ADAM PRACA System is presented in the table below: 


System Name: 

ADAM (Advanced Data Acquisition and Management) 

Point of 
Contact: 

NASA: Randy Segert 

Contract: Margaret Guardia (USA), Susan Ahrens (USA) 

Location: 

Primary system: NASA JSC (Houston) 

Support Sites: NASA KSC (mirror), all SSP sites 

Operational 

Since: 

Initially operational as FRED (Fast Retrieval of Enterprise Data) in 
J994 sponsored by NASA HQ Code Q. It became ADAM in 1996 

Purpose: 

Provide centralized access to SSP PRACA to fulfill NSTS 07700 
trending and analysis requirements. 

Platform: 

on 

Hewlett-Packard V2200, Compaq Proliant 
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O/S: 

HP-UX 11.0.x, Windows NT 4 

Database: 

Oracle 8.0.5.x 

Application 

Programs: 

Client App: HTML interface written m ColdFusion on Windows 
system 

Database Management: Via Oracle tools 

Reporting: Custom Query screens, Crystal Reports, and Hyperion 
Essbase 

Security 

Model: 

Primary: Limited IP access to Web server. Local NT domain access is 
required for the use of Essbase tools. 

Passwords/UID: A password and user ID is required to log into the 
browser front end. 

Requirements 

Document: 

NHB 5300.4 (ID-2), 

NSTS 08126, 

USA 96-dw0001 

Web Info: 

http://usal.unitedspacealliance.com/hq/warehouse/ 

Other: 

ADAM supports many of the other PCASS datasets 


Table 5 - SSP ADAM System Information 


2.2.5.1 Architecture 

ADAM is envisioned to be a central warehouse for all of the SSP PRACA data. Its 
current implementation is as a data store for a copy of each of the separate PRACA 
systems data and schemas. The data gets updated from the source systems daily as a 
complete copy into the ADAM Oracle database. The data warehouse concept is used to 
support future advanced trending and analysis without disturbing the Project PRACA 
systems. Since there is no master database schema or integration capability between the 
copied PRACA databases, the data warehouse is currently only capable of serving as a 
central data store. This is a critical point as this means that any trending and analyses 
must be done on each of the separate data sets and then somehow unified. 

ADAM is a mirrored system, with the central database running on a HP-UX server and a 
Compaq Windows NT system running a web server for client access. One set of 
machines is housed at KSC and another set is duplicated at JSC. 
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Figure 6 - SSP ADAM System Deployment 


ADAM maintains copies of the following SSP PRACA datasets: 

• Orbiter CAR (PDSS) - The Orbiter PRACA Data Support System (PDSS) contains 
data related to Orbiter Contractor-Furnished Equipment, Government-Furnished 
Equipment and Ground Support Equipment for which United Space Alliance (USA) 
has design responsibility. 

• MSFC PRACA - MSFC PRACA contains nonconforming articles for materials, parts, 
assemblies, and systems (items) that are received, manufactured, modified or tested at 
MSFC. 

• KSC IPR/DR/PR - The KSC SPDMS II PRACA database is the repository for SFOC 
(USA), BOC (EG&G), & OMDP (Boeing North American, Palmdale, CA) PRACA 
data. The database also contains Shuttle Payloads IPR's initiated durin g integrated 
operations. The current data set contains the non-arc hived data presently residing on 
the SPDMS computer from approximately l-Jan-1978 thru the last 24 hours. 

• KSC CAAR - The KSC SPDMS II CAAR database contains Corrective Action 
Assistance Request information from various KSC contractors/organizations seeking 
recurrence control action(s) and subsequent closures. Contractors include: USA 
(SFOC), Lockheed Martin (SPC), McDonnell Douglas (PGOC), Boeing North 
American (Orbiter), USBI (SRB), Martin Marietta (ET), EG&G (BOC) and others. 
The data set contains information from 1984 thru the last 24 hours. 

• In-Flight Anomaly - The user can query In-Flight Anomaly data sets using almost any 
combination of this screen's fields. The one constraint to available combinations of 
fields used is when "Choose Data Element 1" is not used, "Choose Data Element 2" 
cannot be used also. 







• Government-Furnished Equipment (GFE) PRACA - The GFE PRACA System 
contains problem data and information related to Government-Furnished Equipment. 

• PR Images (SIMS) - The PR Images (Shuttle Image Management System) screen will 
allow the user to perform an optimized data search across one or multiple data sets. 
The user can select specific data sets (PR Type) to search, enter search criteria for 
commonly used data elements (e.g., problem date, problem status, part name, part 
number, etc.), or select from a pick list or entering in a text specific data elements 
from the selected data sets. Wildcards can be used to support searches. 


2.2.5. 1.1 System 

The Oracle 8.0 database application is housed on a Hewlett-Packard HP Vclass server 
running HP-UX 1 1.0, a Unix variant. A separate Compaq Windows NT system supports 
the web server that maintains a dynamic HTML interface written in Coldfusion. 

Hyperion Essbase software and Crystal Reports software also runs on another Compaq 
for additional database analysis. Both the HP and the Compaq system are mirrored to 
twin systems, with one set of systems located at KSC and the other at JSC. This is for 
data redundancy and to support two separate local infrastructures. The database is 
mastered at the KSC site and maintained and managed at the NLSD USA contract 
building. Client query access is provided via the custom Coldfusion application available 
through the web server running on the Compaq systems. Data storage is provided via 
RAID 5 disk arrays. 

User Access via 



Figure 7 - ADAM Data Warehouse 
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Figure 8 - ADAM System Deployment 

2.2.5.1.2 Network 

Query access to ADAM datasets is open to Space Shuttle Program personnel who are 
on trusted domains in the NASA community. These include, but are not limited to JSC, 
KSC, MSFC, and USA of the other key contractor organizations. Each user is 
authenticated by logging onto the local Windows NT domain; the domain is then 
authenticated at the ADAM Web Server using the Microsoft Internet Information 
Server (IIS) and a ColdFusion front-end application. If the user is not part of a trusted 
NT domain (such as Mac or Unix systems) or outside the local network, a user ID and a 
password are required. 

A diagram of the network connectivity between the two mirrored portions of ADAM is 
shown in the figure below: 
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Figure 9 - ADAM Data Warehouse Network Connectivity 


2.2.5.1.3 Database 

The ADAM database is implemented using Oracle 8.0.x database software on the HP-UX 
system There is no overall data warehouse schema or data field naming convention that 
is consistent across all of the various PRACA system data within ADAM. This means 
that ADAM is a largely unstructured database. 

ADAM does not have a naming standard for entities/attributes within the database. The 
current database design of ADAM is to keep the table and column names m the 
warehouse the same as the source PRACA systems’. The advantage of doing this is to 
minimize the confusion to database users familiar with their transactional systems. Thus, 
a column called ‘colA’ in table ‘tabA’ would map directly from the source system to the 
ADAM data warehouse. With the advent of a web-based interface, column name became 
less important than the definition of that column’s value. It was also originally thought 
that users would feel more comfortable with the validity of the warehouse data and also 
make the ADAM data audit procedures clearer. However, these assumptions have not 
been validated through this Study. 
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2. 2.5.2 Data Collection 


The SSP PRACA data is generated and maintained in individual Element/Project 
databases. On a nightly basis, the new and modified data is extracted from each of these 
systems. A full extract may be done if requested. The data transfer program between the 
source PRACA systems and ADAM is initiated by UNIX shell scripts that are executed 
by a UNIX cron job. The data extract files are electronically transferred to the ADAM 
data warehouse where a loader program is initiated. These programs are a collection of 
UNIX shell scripts and PL/SQL packages and procedures stored in the ADAM Oracle 
database. 

The loader program reads the extracted file and parses the data into mapped database 
table fields. If extensive formatting errors or data errors are detected during the load 
process, the load is terminated and no data updates are committed. Formatting errors that 
are easily corrected are performed and the loader process restarted. Daily reports for the 
loads are generated and distributed via e-mail. If a load was unsuccessful, the data load 
coordinator works with the Project contact until the load is successful, at which time a 
follow-up e-mail is generated and distributed. 


For example, to capture the KSC SPDMS PRACA data, the job is run nightly at 12:10 
a.m. and is explained below: 

a. A UNIX shell script starts the transfer process. The shell script invokes an SQLPLUS 
session that in turn executes the Oracle-stored procedures involved in the transfer 
process. 

b. The stored procedures are stored in the Oracle database residing on ADAM. The 
procedures query the source database for data inserted, modified, or deleted on the 
previous day and subsequently modify the warehouse as appropriate. 
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Figure 10 - SSP ADAM Data Collection Process Flow [S. Ahrens] 


2.2.5.3 Interface 

The interface provided to ADAM clients is through commercially available web browsers 
available for all major computer platforms. The web server application is written in 
ColdFusion as a series of presentation and query HTML-based web pages. The interface 
was developed to provide basic query capability for each of the PRACA datasets. 

The “Search All PRACAs” data screen will allow the user to perform an optimized data 
search across one or multiple data sets. The user can select specific data sets (PR Type) to 
search, enter search criteria for commonly used data elements (e.g., problem date, 
problem status, part name, part number, etc.), or select from a pick list or entering in a 
text specific data elements from the selected data sets. Wildcards can be used to support 

searches. 
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3.0 PRACA System Findings 

Upon completion of the on-site interviews, reviews and technical assessments of the 
multiple PRACA systems, the team identified a set of general findings, and a set of 
findings specific to the four technology areas identified in our Approach. The overall 
assessment of the existing PRACA systems is that they are inefficient and potentially 
vulnerable to data loss and input error. The current approaches do not scale or adapt 
easily to changes. The expert knowledge that is required to utilize the PRACA systems is 
not captured or documented. Overall, the existing PRACA system is incapable of 
supporting Program-level risk assessment. These general and specific technical findings 
are discussed in the following sub-sections. 

3.1.1 General Observations of the SSP PRACA 

Systems 

3-1.2 Quality PRACA Workforce 

The PRACA system workforce is highly skilled and motivated. The PRACA system 
managers exhibit a clear understanding of the important role PRACA data plays in SSP 
safety and have an excellent grasp of the scope and breadth of their system and teams’ 
domain expertise and domain responsibilities. 

During site visits, we typically met with teams of 5-8 people for interviews. From the 
discussions, it was clear that the staff each knew their individual responsibilities as well 
as each other’s skills and team responsibilities. The PRACA system workforce and 
managers freely expressed visions, goals, and plans for system improvements and 
upgrades. It is clear that they creatively could and should contribute solutions for a 
unified SSP PRACA system. 

3.1.3 Current PRACA Does Not Meet SSP Needs 

The PRACA systems in use by the SSP are not sufficient to meet the SSP current and 
future needs (as expressed by NASA SSP management, the SSP requirements 
documentation (i.e., NSTS 07700 Volume XI), and from a sustainable workforce 
perspective). This general finding is primarily due to lack of clarity in expression of a 
vision for PRACA, which is the result of three primary causes. 

1. The SSP is currently able to accomplish its PRACA Shuttle management tasks, 
yielding the perception that “PRACA works.” Because of this perception, 
questions addressing the essential functional, systematic and architectural aspects 
of PRACA are not being asked. 

2. The SSP requirements documentation does not preclude the use of data outside 
the formal PRACA data systems. This enables a capability much greater than 
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would be possible if the users were restricted. to PRACA system data only. As a 
result, the SSP has not been confronted with the data limitations of the current 
PRACA systems. 

3. The current SSP PRACA system is perceived as having a system-wide, user- 
navigable data warehouse capability. This perception is further reinforced by the 
use of domain experts who “extract” data and reports from the various 
Project/Element-designed and supported systems. This gives the impression of a 
system-wide data mining and navigation capability. Because of this perception, 
the SSP is not articulating and advocating a data warehouse re- architecture to 
PRACA (believing it already has one). 

These three reasons are described further in the following sub-sections: 


3. 1.3.1 The Perception That “PRACA Works.” 

The premise of the PRACA capabilities described in NSTS 07700 Volume XI sections 
3.4 and 4.1.5.x is to “integrate program element trend systems, perform analysis, and 
provide data formatted for management visibility” to support Shuttle Safety Assessments. 
The SSP does this by reliance upon subsystem domain experts, and the performing 
SRQ&MA organizations skilled personnel, to report status and trends to the Program 
office The domain experts possess substantial institutional knowledge and are able to 
draw from information outside the PRACA data systems. The reports provided to the SSP 
therefore are based on large amounts of data outside of the PRACA systems. The loss o 
the domain experts and the external data sources would significantly degrade the quality 
of the PRACA reporting and trending. The use of domain experts enhances the quality ot 
the knowledge extracted from the PRACA systems, giving the Program Office the 
mistaken impression that the PRACA systems alone possess equivalent data and 
knowdedge-generation capability as that presented in the reports. 

For the trending and analysis groups to perform their tasks using current PRACA, it is 
sometimes necessary to filter or correct the data and generally augment PRACA data to 
produce meaningful results. Several NASA SR&QA groups identified this practice as 
necessary to perform meaningful trending and analysis for the Program office. Th e JSC 
Shuttle Orbiter (Code NC) SR&QA group, for example, created and manages the Shuttle 
Risk and Reliability Analysis Database (SRRAD) and created manual and automated 
filtering and data processing routines for this purpose. The SRRAD database and data 
filtering and correction process are examples of data and expertise upon which the SSP 
reports are dependent but are not part of the formal PRACA system. 

3.1. 3.2 The Use of Non-PRACA Data in Reports 

The use of data outside the formal PRACA system has allowed the evolution of informal 
data systems upon which the SSP depends. These informal databases exist, m many 
cases, at the expense of a proper upgrade of the formal PRACA systems. 
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The base SSP PRACA requirements documentation (NSTS 07700) imply that a “PRACA 
System” will be the official repository for all data necessary to manage Shuttle problem 
reporting and corrective action assessment, diagnosis, and trending. However, the NSTS 
08126 Revision G requirements also tum responsibility for generating the status, 
assessment and diagnosis over to Project/Element domain experts with no constraints on 
the data they are permitted to use to generate the reports. In order to produce the best 
reports possible, the experts naturally draw from all the information they can access. 

The dependence of the SSP on domain experts to produce reports is codified in the 
requirements document (i.e., it is the responsibility of the domain expert centers to 
manage their subsystems and report their data to the program office). This has effectively- 
hidden the lack of stand-alone capability the SSP desires for PRACA. It has also hidden 
the fact that the PRACA system cannot be data mined by the non-expert users. The expert 
users can manually navigate the various information systems using implicit and innate 
familiarity with the data, thus giving the appearance of a data mining capability. A 
neophyte (or non-expert user) would not have the knowledge or context to understand the 
syntax, or codes used for each of the PRACA systems. Indeed, an expert on one SSP 
PRACA system is not an expert on all PRACA systems. 

As with any large requirements-based information system, the domain experts have found 
that it is easier to build and manage an informal database to augment PRACA data 
“outside” of the SSP PRACA purview' than it is to upgrade the SSP PRACA databases. 
For example, as mentioned in the section above, the JSC SR&QA group performs 
trending and analysis for the Program office by creating and managing the SRRAD 
database. The SSRAD database is used in conjunction with the PRACA data but is not 
part of any formally recognized PRACA system. These adjunct PRACA data sources 
and their requirements need to be addressed in the overall SSP PRACA picture. 

3. 1.3. 3 The Perception that PRACA is a Navigable 

Warehouse of Data 


The SSP has expressed wishes to incorporate the individual PRACA systems into a 
unified program data warehouse but has not initiated a major re-architecture or training 
program to elevate the focus from the SSP Element/Project level to the Program level. 
The Project-level or component PRACA systems resident at JSC, KSC, and MSFC have 
been designed and implemented by the PRACA domain experts at each center. These 
systems were developed to meet their individual task and project support needs as stand- 
alone systems. The inevitable local prioritization of tasks results in no two systems 
having the same global motivation for existence or the same technical implementation 
basis. The Project-level systems are not innately amenable to a hybrid data w'arehouse 
architecture. 

USA’s ADAM is a good first attempt to provide unified PRACA data and some 
associated information. The ADAM effort sidesteps the individual Project PRACA 
system differences by duplicating the data onto one site, where an interface and mapping 
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can insulate the user from the navigation difficulties. ADAM cannot, however resolve 
the issues of data quality, consistency, integrity, and breadth, which are limited by t e 
source PRACA systems. ADAM is increasing data visibility and bringing attention to 
data errors and data field mapping inconsistencies across the PRACA systems. ADAM is 
not addressing the system-wide architectural changes required of the Project systems. 


Much of the data the Program requires and desires from the PRACA systems to make 
them more useful for Safety and Risk analysis are not consistent with the current 
Proiect/Element focus. Data currently gathered for the Program that is not necessary for 
local domain needs is generally handled with less care because its value and use are 
locally subjective. This is important to note for a number of reasons. It can lead to 
incorrect entries because the person making the entry does not understand the purpose of 
that data field in the PRACA form. Additionally, end users such as people doing 
trending and analysis may not understand the context in which the data were acquire an 
therefore why fields are being filled out in particular ways at operational sites. These 
mutual misunderstandings can produce incorrect data, which degrades the validity of any 
trending analyses using these data fields. The main commonality between the PRACA 
systems comes from the NSTS.08J26 document. While each of the various PRACA 
systems comply with the current 08126 revisions, the Project-level PRACA systems have 
there own guiding requirements and documentation, of which the 08126 requirements 
play only a partial role. Data field naming conventions are solely left to the Project to 
implement in their local databases. 


Streamlining efforts (efficiency, turn-around time, work-flow) at the Project and domain 
level tend to resist addition of fields that impede the streamlining efforts. This is due in 
part because the local PRACA system is often used for a number of functions besides 
problem reporting and corrective action. Thus many individuals collecting the data may 
not see their job as connected to the SSP system-wide PRACA effort. This also 
contributes to opacity and misunderstandings of the roles and purpose of the SSP 
PRACA systems. 


3.1.4 No “PRACA System-wide” Philosophy 


There is a lack of a clearlv expressed vision for SSP PRACA and subsequent lack of a 
system-wide buy-in to the “PRACA System” philosophy. While the Space Shuttle 
Program expectation for PRACA is as a part of the PCASS, many view the PC ASS as 
one of the several systems comprising PRACA data sources rather than as a PRA en 
design goal. The problem is further complicated by a lack of clearly acknowledged 
“PRACA owner” For example, none could be identified or established during a \isit to 
the SSP office in January 2000. (Note: Since January a PRACA owner has been 
identified, within the SSP as well as for the USA contract, by the JSC PET activity). 

The creators or Project-level managers of the PRACA systems accept the SSP system- 
wide data resource goal but do not yet view PRACA data as a Program-wide resource 
that can be controlled and managed by the SSP office. An overall “PRACA System as 
an organizational or technological entity does not exist, and was not required by the 
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NSTS 08126 Revision G document. The JSC PET has rewritten and enhanced the NSTS 
08126 document to a Revision H and this is d: ussed in section 3.1.4. 

3. 1.4.1 Effect on PRACA Data 

The data collection workforce is not currently trained in PRACA system Program data 
needs and usage. The data collection complexity imposed on the local (domain) PRACA 
teams is not a primary consideration addressed in the PRACA requirements 
documentation. Because of the Project-focused development of the individual PRACA 
systems and their work practices, the resulting multiple definitions, levels and functions 
of PRACA lead to opacity between parts. That is, each user of a PRACA system 
understands it from the viewpoint of their Project/Element and their task. There is not a 
training process enabling a universally shared understanding about what PRACA data is, 
what its levels are, who has responsibility for which parts of PRACA data, and how to 
weigh the priorities of Program data collection against local task schedules and deadlines. 

We found that Project users, the engineers and technicians who originate reports, were 
more likely to understand PRACA data from the perspective of their center, or even of 
their specific job and its procedures, without understanding the larger implications for 
how the system(s) works and the SSP-wide service that it provides. This can lead to 
tensions between the functions, and to the possibility of bad data being entered into the 
system. 


For example, technicians at KSC who have found a nonconformance are required to fill 
out a PRACA Problem Report, Interim Problem Report or Discrepancy Report as a part 
of the process for dealing with nonconformances. However, the report’s central use for 
them is to schedule the work of repair and assignment of safety constraints on other work, 
which may not be performed until the nonconformance is dispositioned. Many of the 
required data fields in the problem report form that are relevant for problem reporting are 
irrelevant to the scheduling process, and may be seen by users as demanding information 
to which they do not have ready access or which requires too much valuable time to 
answer. 

One example of such a field is “Vendor”. It may be useful for developing trending data 
at the Program level, but this use is not clear to technicians. The field is not necessary for 
any of the work that the technicians do, nor is it particularly valuable to the engineers" and 
schedulers who direct the technicians work. This confusion can lead to technicians 

filling in any value that they know will pass the inspection process, rather than attempting 
accuracy. 

3.1. 4.2 Effect on Future Potential for PRACA 

The lack of a clearly articulated and adopted vision across the PRACA systems has an 
effect on the future potential for safety and risk analyses. Many of the visions expressed 
by NASA senior management for the ideal “PRACA System” include a prognostic 
capability with data search, navigation, and mining that extends across the Shuttle 
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Program. Without some effort to promote unity amongst the systems or their data, the 
potential technical leverage from similar systems (e.g., the Aviation Safety Program) and 
NASA research Programs such as Design for Safety will be reduced. There are many 
opportunities for interaction and data sharing with the digitized Shuttle components 
databases, commercial aviation maintenance planning and scheduling systems, 
model-based reasoning systems that could significantly enhance the utility of the PRACA 
data for safety and risk analyses within the SSP. This is discussed further in the 
recommendation section of this report. 

3.1.5 NSTS 08126 Revision H Updates 


The JSC PET has completed much of a rewrite and update to the NSTS 08126 document. 
This revision is appended as Revision H to identify it as the successor to R e ' lsl0n 
Revision H better reflects the desired scope and global functions expected of the SSP s 
PRACA system. The Space Shuttle Program Review Control Board is expected to 
approve 08126 Revision H in the summer of 2000. 

Revision H now states that the goal of the PRACA system is to establish a process to 
continuouslv improve the safety and reliability of Space Shuttle hardware software, an 
critical ground systems. The PRACA system will provide the SSP and all SSP 
Elements/Projects: 

1) Accurate and immediate visibility into problems; and 

2) An accurate historical database to support problem trend analysis, provide failure 
history, support anomaly investigation, and to document corrective actions. 

Revision H also recognizes that PRACA is only useful if the reported information is 
accurate and correct. ^It emphasizes that sufficient attention must be paid to insuring 
accuracy of the data comprising the problem report, failure summary, root cause analysis 

and in/out-of-family screening. 


NSTS 08126 Revision H defines and enhances the SSP requirements for problem 
reporting;- analysis, disposition, resolution, and trending. Problems that are documented in 
PRACAinclude: Space Shuttle hardware (Orbiter, GFE, Flight Crew Equipment, SSME, 
ET, SRB, RSRM, and cargo integration hardware), Orbiter software discrepancies, 

SSME software discrepancies, Launch Processing System (LPS), Ground Support 
Equipment (GSE) and Launch & Landing (L&L) facilities that support mission to 
mission processing of flight hardware. The Revision H document establishes: 


1) Uniform criteria for reporting problems; 

2) Requirements for problem disposition and closure; 

3) Requirements for documentation of corrective action, 

4) Requirements for problem documentation to support engineering and trend analysis; 

5) Requirements to support logistics management, and 

6) Definition of problem report data elements and terminology. 
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The NSTS 08126 Revision H now addresses the requirements for extracting the data 
necessary for trend analysis and reporting. In addition, this latest version of the PRACA 
system requirements greatly improves and clarifies the requirements for the PRACA 
systems. Work is continuing on the PRACA data element definitions and establishing the 
database code translation tables to enable some mapping between the various PRACA 
systems data. These definitions and tables are to be included in the final version of 
NSTS 08126 Revision H. 

3.2 PRACA Assessment Area Observations 
3.2.1 User- Interface Findings 


The team interviewed the various users and evaluated the PRACA systems interfaces 
where possible. In the case of the JSC government-funded equipment system and the 
KSC group support system the team did not achieve hands-on access, due to the teams 
remote location, exclusion from the firewall-protected LAN (security model) and the user 

interface being designed for local access OjUx. 

The team made the following observations with regard to the user interface: 

• There is no SSP user interface design specification document. 

There is no reference user description (i.e., who is the interface designed for, 
what skill level, what resources are at the users disposal, etc.) 

— There is no reference task list (i.e., what functions does the user need to 

perform, what data does the user need to access, what ways does the user need 
to see the information formatted, etc.) 

~ There is no reference host/client platform or Operating System target for the 
interfaces. 

• Each of the PRACA systems has a unique user interface. These interfaces 
demonstrate varying degrees of complexity and capability ranging from command 
line SQL to Web based interactive reporting. 

- Most of the interfaces are implemented in custom software. 

- None of the systems are fully Web enabled, though ADAM and PDSS are 
moving in that direction. 

- Most systems have Graphical User Interfaces (GUI’s); the remainder use 
command-line mode text interfaces. 

• All the User Interfaces provide a data query capability. 

• All the User Interfaces provide varying degrees of reporting and trending capability, 

A summary of comparison information for the various PRACA systems User Interfaces is 
presented in the following table: 


PRACA 

Data Query 

Report/Trend 

Client Application 

Report/Trend 

System 

GUI(G) or 

GUI(G) or 


Group Support 


Text(T) 

Text (T) 
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PDSS 

G 

G 

Custom Windows App. 

Y (JSC Code NC) 

GEE 

G 

G 

MS Access run-time 



KSC 

T 

T 

Custom/SQL 

Y (USA contract) 

MSFC 

G 

G 

Browser 


ADAM 

G 

r- G 

Browser 



Table 6 - User Interface Comparison 


A further comparison of the user interfaces is presented in the following sub-section. 

3.2.1. 1 JSC’s PDSS 


The JSC PDSS User Interface (UI) is a Windows-based custom interface with analysis, 
reporting and trending capability. The interface is very intuitive, with icons to launch 
applications, pull down lists, select lists, and script generating and editing capabilities. 
The PDSS UI has a large external user base and the support group incorporates user 
feedback in ongoing software upgrades. 


The following figure shows the PDSS database query window: 


PDSS * OATA SELECTION SCREEN/ 'WHERE* CLAUSE SEHINU 


■ * ■- - ' 11 999 - 05 - 30 ] 



Figure 11 - PDSS Query Screen [T- Dinh] 


The PDSS UI allows plotting of the data to produce trend analysis. The software for this 
is called Trendtool and is a custom application provided to users by the PDSS support 
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group. The following figure shows Trendtool’s plotting capability (count vs. data type) 
and also displays the selection criteria for the advanced user: 
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Figure 12 - JSC Orbiter Data Reporting and Trending Interface [T. Dinh] 


The PDSS support group has done an excellent job developing this interface and the 
attention to customer feedback is clearly evident. 

3.2.1. 2 JSC’s GFE 


The JSC GFE User Interface is a Windows-based Microsoft Access interface with 
analysis, reporting and trending capability. The trusted client machines access the GFE 
database by executing a run-time version of MS Access. The data can be queried and 
reports generated from either the Host or the authorized client computers. Database 
management is performed on the Host computer via MS Access directly. The interface is 
very intuitive with icons to launch applications, pull down lists, and select lists. The GFE 
UI has an external user base. 

The following figure shows the GFE database query window: 
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Figure 13 - GFE PRACA Access Database Interface [S. Ferguson] 

The MS Access GFE UI allows for report generation and a basic plotting capability is 
also possible. The following figure shows the GFE UI reporting capability. 
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Figure 14 - JSC GFE Data Reporting and Trending Interface [S. Ferguson] 

3.2.1. 3 KSC’s SPDMS 


The KSC SPDMS User Interface access is via command-line to the IBM VM mainframe. 
Users are connected to the mainframe via a terminal session. The figure below shows a 
sample of the report output from a command line query' to the system: 















Figure 15 - SPDMS Command Line Query Output 


There is also a variant of the ADAM custom user interface to access a copy of the 
SPDMS mainframe system PRACA data that resides within the ADAM data warehouse. 
The ADAM interface to the SPDMS data is shown in the figure below: 
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Figure 16 - ADAM Variant SPDMS User Interface 


3.2.1. 4 MSFC’s PAS/UP RAC A 

The MSFC PAS provides an interface to the UPRACA system and PRACA data via a 
web-based access using any of the commercial browsers as the client front-end. The user 
interface was developed for the S&MA office and expert users. It is not developed as a 
general purpose or managerial support interface to PRACA data. The web-based online 
application is written in C that generates the HTML used for screen displays. Embedded 
SQL is used in the C program for the database calls. The interface is intuitive with pull 
down lists, and select lists. The MSFC UI has an external user base. 

The following figure shows the MSFC database query window: 
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Basic reporting is also available via select lists in the interface shown above. 

3.2.1. 5 USA’s ADAM (IPAS/WebPCASS) 

The user interface provided to ADAM clients is through commercially available web 
browsers available for all major computing platforms. The web server application is 
written in ColdFusion as a series of presentation and query HTML based web pages. The 
interface was developed to provide basic query capability for each of the PRACA 
datasets. 

The “Search All PRACAs” data screen will allow the user to perform an optimized data 
se^ch across one or multiple data sets. The user ean select specific data sets (PR Type) to 
search, enter search criteria for commonly used data elements (e.g., problem date, 
problem status, part name, part number, etc.), or select from a pick list or entering in a 
text specific data elements from the selected data sets. Wildcards can be used to support 

searches. 

The following figures show the ADAM interface and the database query window: 
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Figure 18 - ADAM User interface 



Figure 19 - ADAM Cross PRACA Query 
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The following figure shows an example of ADAM’s trend plotting capability: 



ADAM’S User Interface is a good example of a web-enabled and standardized interface 
across multiple databases. The ADAM, and MSFC UPRACA, have some degree of 
commonality in their design but the implementation of navigation is different. 
Unfortunately in order to query the PRACA source databases directly a user would need 
to know and be familiar with all of the PRACA systems UIs. 

3.2.2 Database and Data Management Findings 

The team interviewed the various users and evaluated the PRACA systems databases 
where possible. In the case of the JSC government-funded equipment system and the, 
KSC <*roup support system the team did not achieve hands-on access, due to the team s 
remote location exclusion from the firewall-protected LAN (security model) and the user 
interface being designed for local access only. 

The team made the following observations with regard to the databases and data 
management: 

• The PRACA systems have fairly common approaches to database architecture and 
implementation: 

- All of the databases are relational with “report number” (or its field name 
variant) being the most common field across tables. 

- The smallest architecture is based on 12 primary tables. 
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Commercial off-the-shelf (COTS) database applications are used, with Oracle 
being the most common application chosen. 

• The databases are deployed on PC Compatible computers running the Window's NT 
OS or on Unix (Sun or HP) systems. SPDMS runs on an IBM mainframe computer. 

• None of the databases are currently taxing system performance. 

- JSC’s PDSS is the largest (known) database at -5 GB 

- JSCs GFE is the smallest at ~10 MB 

• The oldest systems hold data that was created in 1974. 


A summary' of comparison information for the various PRACA systems Database 
Implementations is presented in the following table. 


PRACA 

Database 

Application 

Vendor 

System OS 

Host System 

DB Size 

Date created 

PDSS 

Oracle 7 

NT 4 

PC compat 

-5 GB 

-1974 

GFE 

r MS Access 97 

NT 4 

PC compat 

~10 MB -1974/1998 

KSC 

SQL/DS 

VM 

IBM 9000 

unknown -1978 

MSFC 

Oracle 7 

Solaris 2.6 

Sun 

-230 MB -1978 

ADAM 

Oracle 8 

HP-UX 

HP V2200 

NA 

-1994/1996 


Table 7— Database Comparison 


Overall, the PRACA systems lack formal documentation to support the design and 
implementation of all five subsystems, including ADAM. This means either a partial or 
total a lack of entity relation diagrams, data dictionaries, and database requirements 
documents. Some systems, like the MSFC UPRACA, were better documented than 
others. 

3.2.2. 1 JSC’s PDSS 

The Orbiter database was originally implemented in DB2. The PDSS utilizes several 
process models and a single database. The Orbiter PRACA process models include 
process flows for the initial CCAR/TCAR decision and the subsequent processing of 
these problems. There are twelve primary database tables with several additional tables to 
support applications processing, codes, and general database support. There is one entrv 
in the common table for each problem record. There are repeating entries in five other 
tables for each problem record. There are six tables that contain narrative text 

informauon. The field JSC_REPORT_# is the unique identifier for each problem record 
and exists in each table. 


JSC’s PDSS 12 primary table reladonal architecture is shown in entity relation diagram 
form below: ' ~ 
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Figure 21 -NASA JSC Orbiter Database Schema (Original DB2 Architecture circa. 1988) 


3.2.2. 2 JSC’s GFE 

The GFE database was originally part of the Orbiter PRACA system. Like PDSS the 
GFE data has followed the Space Shuttle development,with the earliest data m GFE 
originating in 1974 when PDSS was implemented in DB2. The GFE data were separated 
outof Orbiter PRACA during the PDSS Oracle upgrade in the 1997-98 time frame. This 
upgrade left two separate data systems (GFE and PDSS) resulting in separate data 



management responsibilities. JSC’s GFE 12 primary table relational architecture is 
legacy from the original DB2 design and it shares this common architecture with PDSS. 
The MS Access .mdb database tables view form is show'n below: 



Figure 22 - JSC GFE Database Schema [S. Ferguson] 


3.2.2.3 KSC’s SPDMS 


A relational database SQL/DS is used on the SPDMS to manage the PRACA data at 
KSC. The SPDMS holds at least nine separate databases for various group operations 
functions. For the PRACA database, there are 35 primary and support tables. Common 
fields are “Reference_Report_Number” and “USERJD”. KSC’s SPDMS VM relational 
architecture is larger than the others and requires three figures to present all the tables. 

The main figure (1 of 3) is shown for comparison to the other database designs in the 
figure below: c 
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Figure 23 - SPDMS Database Schema (1 of 3) [D. Mondshein] 


3.2. 2.4 MSFC's PAS/UPRACA 

The MSFC Problem Assessment Systems UPRACA is based upon the Oracle 7 database. 
It has approximately 18,000 problem reports and in excess of 900,000 records . ■ 
sixteen table database is approximately 230 megabytes in size, with the whole IJPRACA 
database able to hold up to 5 gigabytes of data. All tables share the common field 
“MSFC_REPORT_#” . The IJPRACA relational architecture is shown in entity relation 

diagram form below: 
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Figure 24 - MSFC Launch System Database Schema 


3.2.2.5 USA’s ADAM (IPAS/WebPCASS) 


The ADAM database is implemented using the Oracle 8.0.x application software on the 
HP-UX system. There is no data warehouse schema or data naming convention that is 
consistent across all of the various PRACA system data. Ideally, all data relationships 
should be correlated and cross-referenced; however within ADAM there are no keys to 
show the relationships between the data fields. The ADAM design documentation lacked 
any type of keys (primary and foreign). Due to the lack of a unified PRACA data naming 
and database schema from all of the separate PRACA systems, ADAM is largely 
unstructured and very limited in capability. 

The original intent of ADAM was to act as a data warehouse that would have been able 
to take end-user’s queries against all of the PRACA data and retrieve the associated 
detailed results. According to our findings, ADAM is a front-end transaction traffic 
controller that directs user’s request based upon a fixed prefix entity identifier via the 
database mechanism (e.g., physical infrastructure of the entity and property 
implementation). 
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ADAM does not have a naming standard for entities/attributes within the database. There 
is no unified or overall database schema for the various PRACA tables within the system. 

The entity relationship (E/R) diagrams that were provided for ADAM were done in 
unfamiliar E/R standard notation. Each of these diagrams indicated unintentional flaws in 
the structure. Currently, each of these subsystems is comprised of entities and properties, 
but no relationship nor subtype. 

A metaphor for the current state of the ADAM data warehouse is as follows: Imagine 
several teams of people each providing several bound sheets of paper, of various sizes 
and format, written in a particular language. Taking all of these bound groups of 
and unbinding them, and stacking them into a book is equivalent to the state of AD 
and the Project PRACA data currently. The book that represents ADAM has no table of 
contents, no index, and each sheet of paper is of different size with a different language 

written on it. 

ADAM developers have recently started populating a data dictionary (a common 
-language in the above metaphor) in ADAM that contains descriptions for the data in the 
individual datasets (sheets of paper) and more importantly, how the data from the 
different datasets relate to one another (an index and table of contents). A front en or 
this data dictionary is not yet available. Establishing the commonality between the data 
sets was very hard and has yet to be validated. ADAM can cross-reference a KS 
PRACA “part_prog_no” column to a PDSS “partno” column, but if the two systems are 
not using standardized part numbers as source data then the query is fruitless. Although 
part numbers are not the ideal example, there are a lot of other data that is non-standard 
and poses similar problems. 

3,2.3 Network and System Architecture Findings 

As described in the Approach section, the team evaluated the System Architecture by 
considering several aspects including: 

• Server technology 

• Network security implementations (firewall, proxy, trusted host, etc.) 

• Center-to-center communication techniques 

• Technology maturity levels relative to state-of-the-art. 

The team made the following observations with regard to the Systems and Networks: 

• The accesses to the PRACA databases are via PC Compatible computers running 
Windows NT or commercial UNIX systems (Sun or HP). 

• None of the database administrators report that they are taxing system 
performance. 

- The disk space requirements are relatively low. 

- The query speeds are satisfactory (as evaluated by the user community). 

• All of the databases are networked. 


- It would be possible to connect to any of the databases from an Internet- 
connected computer 

— The firewall implementation provides access restriction but the security 
effectiveness is not clear. 

— There is no SSP document that identifies access restriction design 
requirements. 

- There are no System level processes to monitor for security violations. 

• All of the systems have upgrade potential to state-of the art capability. 


PRACA 

Database 

Host 

System/OS 

DB Size 

Network 

Security 

Password 

UID 

Technology 

Maturity 

(Low-Med- 

High) 

PDSS 

PC com. 
NT 4 

-5 GB 

LAN 

Firewall 
Trusted client 

Write: Yes 
Read: no 

Med 

GFE 

PC com. 
NT 4 

-10 MB 

LAN 

Firewall 
Trusted client 

Read: No 
Admin: Yes 

Low 

KSC 

IBM 9000 
VM 

Unknown 

LAN 

Firewall 
Trusted client 

Yes 

Low 

MSFC 

Sun 

Solaris 2.6 

-230 MB 

LAN 

Firewall 

Yes 

Med 

ADAM 

HP V2200, 
HP-UNIX 
PC com. 
NT4 

NA 

LAN 

Firewall 

Yes 

Med 


Table 8 - Network and System Comparison 

3.2.3. 1 Security Issues 

There are several security issues associated with the current networked implementations 
that are noteworthy: 

• It would be possible to access any of the databases from an Internet-connected 
computer. 

- The firewall implementation provides machine level access restriction but the 
security effectiveness is not clear, (e.g., trusted client without user 
authentication does not preclude unknown user access.) 

• There is no SSP document that identifies data security requirements and access 
restriction design requirements. As a result, neither the purpose of the security 
measures nor their effectiveness can be assessed. 

- The measures could be ineffective. 

— The measures could be unnecessary. 

• There are no system-level processes to monitor for security violations. 

- There may be ongoing security breaches that are not being tracked. 


69 




• A random check of web site implementations was noted to provide potential 
security holes: 

_ http://www hnnston.ssd-bna.boeing .com/d331/pac/ is open to the public and 

gives an index listing. . . 

- http://usal ■iinitedsDacealliance.com/hq/ also gives a listing but it is not a 

public page. 

3.2.4 PRACA Work Process Findings 

To understand the PRACA data life-cycle (how it is created, used, stored, managed and 
updated) and the interrelationship with the human in the work process, a work process 
study was performed. Some of the findings of that study have been used in previous parts 
of this report. This section discusses two important general findings from the work 
process study (limited to the KSC PRACA data collection, and JSC Orbiter problem 

reporting closure processes): 

1. The dependence of PRACA on people and the paper data record; and 

2. The importance for PRACA of human factors and organizational issues. 

3.2.4.1 Key Role of Paper and People In PRACA Work 
Process 

PRACA is normally thought of as a collection of databases: however, the distributed 
nature of those databases, each managed by different groups, appears to contribute to a 
continuing supply of paper flowing through the system. Paper representations play the 
key role in the information flow process at several points. While this might not be 
surprising at the initial input stage, there are significant uses of paper at many subsequent 
stages of the problem reporting process, which are probably sub-optimal Also, it is 
apparent that in more than one of these situations, the PRACA system relies heavily on 
key personnel to manage the paper trail and determine its disposition and follow up on 

the process. 

This extensive use of paper (and supporting personnel) in the later stages of the process is 
important for two reasons: the cost of printing, filing, and transporting the paper records 
and the issue of accuracy due to transcription errors. There is always a possibility of 
error when data is re-keyed into the system. For example, while there is an electronic 
transfer of information from Huntington Beach to the Orbiter database, some information 
is also transferred to a web site and must be re-keyed. 


3.2.4.1.1 Use of Paper In Initial Filing of Problem 

Reports 

The following is a description of how an initial problem report is filed at KSC during 
Shuttle processing, with an emphasis on the use of paper: 
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Initially, a nonconformance is discovered, usually by a technician in the course of their 
work, but sometimes by an inspector. The person discovering the nonconformance 
begins by filing an Interim Problem Report, Problem Report, or Discrepancy Report. 
Note that these are all filed using the same paper form; they differ only in the box that is 
checked for Field 1 : Report Number. The form may be filled out directly on a computer 
located at the Test and Inspection Record (TAIR) station, but more often is filled out on 
paper. In general, technicians do not carry blank forms with them. They usually jot 
down the required information on a piece of paper they have with them, then go to the 
TAIR station, where they either fill out a paper form (KSC Form 2-151 shown below), or 
more rarely, directly on a computer-based form. We have been told approximately c ' % 
of these forms are filled out manually, since many technicians are not comfortable i g 
a keyboard, and prefer to use the paper form. 
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Because the form is usually not filled out in the physical proximity of the problem site, 
the technician may have to make several trips back to the problem location to g. 
information omitted in the first round of note taking on the nonconformance. 1 e are a 
number of fields which are particularly likely to be forgotten or misidentified. Tnese 
include: 

- 4. End Item Control Number; 

- 10. FSCM/Vendor; and 

- 11. NHA (Next Higher Assembly) 

(Note. These candidates for particularly error-prone fields come from interview data. 
Additional observational data would support or reject these candidates and suggest 
possible further problem fields.) Because the TAIR station is physically located at a 
considerable distance from the Shuttle itself, although within the same building, each ip 
back to get the required information to fill in a forgotten field demands significant tirr 
for the technician to climb in and out of the Shuttle and then travel up and down three 
flights of stairs to the TAIR station. 



Figure 26 - KSC TAIR Station 


The person discovering the problem fills out the relevant items for fields 1-17, and then 
gives the form to the TAIR station. TAIR station personnel enter the data. They then 
initiate a scheduling process, which requires an engineer to determine the potential 
constraints imposed by the nonconformance, criticality of the nonconformance, and 
disposition/corrective action for the nonconformance. Engineers and inspectors work 
with paper copies of the form, and signatures and inspection stamps located on the paper 
form represent their decisions. (Note: There is an inspection process for repairs. In 
addition there is an inspection process for the Problem Report forms. Additional 
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observational study could determine exactly how these processes work, and what their 
impact on PRACA data is.) 

When the nonconformance has been dispositioned and the form completed, the final step 
before the report is closed is a code check, which is represented by field 34 of the 
Problem Report form. This is done by a USA engineer, who determines whether the 
document was processed correctly. This check is represented by 10 digits, which code 
proper filling out of specific fields in the form. 

3. 2.4. 1.2 Paper Processing by the S8P 


John Mulholland, Deputy Manager for Operations in the Space Shuttle Vehicle 
Engineering Office, is an ultimate decision authority in the PRACA system(s). Mr. 
Mulholland uses CARs (responding to PRACA PR's) to n^e go/ncngo d«is.ons on 
Shuttle flight safety that are based on issues of criticality, specifically Cnt 1 an 
2 items. It is interesting the extent to which his use of the PRACA data relies on paper: 
printed-out database forms, printed out emails, yellow Post-It™ Notes, etc. This section 
explores the reliance on paper in this process. 

When a complete failure analysis report is assembled, the Subsystem Manager (SSM) 

and the Problem Resolution Team (PRT) use it to create a problem disposition to a 

Corrective Action Report (CAR). (Note: Further fieldwork could determine how much of 
this report involves the physical assembly of pieces of paper, reports, test outcomes etc., 
which are then attached to the CAR and where those reports are ultimately filed.) Once 
all of this information is assembled, it is sent to the Boeing Problem Action Center (PAC) 
office in Huntington Beach. They verify that all information is included, and the package 
is sent electronically to the local Boeing office at JSC. That office makes a i hard 
copy/copies of the report material, which has also been entered into the PDSS database. 
At the local Boeing office, personnel also load much of the information to a website, a 
process that requires some re-keying. The hard copy/copies are then separated out for 
circulation to required engineers and personnel at various sites at JSC, whose signatures 
are required to close out the CAR. 


In this process, the system relies on key personnel to sort the material, make decisions 
about who should get what information, and physically walk the matenal through the 
system. CARs will be separated out and distributed to responsible engineers, whether at 
NASA in the Engineering Directorate or at USA, System Area Managers (SAMs). 
According to criticality, they will be taken to Mr. Mulholland in the Orbiter Project 
Office. The documents are placed in folders according to a prescribed color code, and 
physically walked or driven to the appropriate offices by support staff. 

Since Boeing has begun loading the material to a website, some engineers have taken the 
opportunity to review the material on-line before the hard copy arrives on their desk. This 
action is speeding up the review process somewhat. However, the time-consunung, 
physical process of driving/walking these copies from the Boeing office to the USA 
office and to other sites in various buildings at JSC is still in use. 


If Mr. Mulholland approves the problem resolution, he signs a paper copy of the form. 
The documentation package gets picked up by staff from the local Boeing office and is 
hand earned back to that office where the information is entered into the system and the 
CAR is closed. 

If Mr. Mulholland does not approve the report or he has questions for the PRT or SSM, 
he generally indicates that on a yellow Post-It™ Note and places the hard copy back into 
the system by placing it in his office outbox, where it gets physically picked up by 
Boeing personnel and taken back to their office. If Mr. Mulholland is able, he will often 
call the SSM with the question, but this is not always possible, so the hard copy must go 
i; - back to system with his questions : cached. Once back in the USA/Boeing office, support 
staff will email the appropriate engineer with Mr. Mulholland’s question. That email and 
any replies will be attached to the package as hardcopy attachments, at which point, the 
whole package is then physically taken back to Mr. Mulholland for his approval and 
signature. 

~3;2.4.2 - -Types off Work Practice Issues 

In our initial study of the PRACA work practice (limited to the KSC PRACA data 
collection, and JSC Orbiter problem reporting closure processes) we found a number of 
problems and areas for improvement. These fall into three categories: Work Practice and 
Human Factors issues, Organizational issues, and Technical issues. We discuss these in 
turn. 


3. 2.4. 2. f Work Practice and Human Factors Issues 

The KSC technicians and engineers have not received training in what PRACA is, or 
does, nor in what kinds of information are being requested for PRACA and why. The 
PRACA purpose is learned as on-the-job training in how to fill out a PR form. The 
trainee is usually most concerned with two things: how to fill out the form so the work 
will be completed correctly; and how to fill in the form fields so that the form will not be 
rejected by inspectors. As mentioned above, this can lead to an apprenticeship in 
learning what kinds of information will be accepted, rather than in learning how to 
provide accurate information. 

In general, KSC technicians and engineers do not fill out a PR form at the site where the 
problem is located. This can lead to several trips back and forth from the problem site to 
determine part numbers, exact location of problems, physical sketch of problems, etc. 
Inaccuracies can be introduced in every trip. 

There are a number of local differences in how different organizations fill out Problem 
Report fields. Some of these differences are directly relevant to the work of that group, 
while some of the differences are historical. For example, some groups use a hazard 
assessment stamp for block 30. Engineers must initial and check this. Use of this stamp 
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is not necessary for the final report. Rather, it is a local adaptation that allows for the 
establishment of safe conditions during repair. 

Some groups describe nonconformances on a problem report using the syntax: Is X 

“Should be Y”. Other groups simply rely on the heading of the field to indicate that the 
descriptions refer to the problem and its suggested disposition. These differences are not 
important in themselves. However, they could become quite important in an attempt to 
do natural language understanding of the historical backlog of problem descriptions and 
dispositions. (At least one such project is underway as collaboration between KSC and 
Central Florida University.) 


Local names for parts and materials may be different from the names required on the 
Problem Report. For example, at KSC, technicians may describe tile material as 
“Nomex ” the generic name for the material. The required terminology for the report is 
“FRSI,” referring to Nomex that has received specific treatment to serve as Shuttle tile 

material. 


The PRACA PR must include a page count of how many pages are contained in the 
associated documentation. However, there are ambiguities in whether appendices should 

count in the page count. 


3.2.4.2.2 Organizational Issues 

We have learned that different organizations may categorize problems differently. For 
example, the Palmdale facility is said to categorize some wiring problems as fair wear 
and tear,” which KSC would treat as non-conformance requiring a problem report. 

During the interviews, it was suggested that one measure of organizational accountability 
is the number of problem reports filed. This would tend to create a climate where 
reducing the number of problem reports filed, by tending to identify a nonconformance as 
a less significant category, has incentive. This is an important issue for further 
investigation 


3 . 2 . 4 . 2.3 Technical Issues 


PRACA assumes a strict hierarchy of problems, based on the tree structure of the Shuttle 
assembly. This makes it difficult for the inspectors to document or describe problems 
that result from interactions between components in different assemblies or systems. 
Note that two components may be distant from one another on the tree-structured 
representation of the Shuttle’s part, while being within close physical proximity, and 
hence liable to physical interaction. 


The TAIR station contains all the documentation of any kind that travels with a Shuttle as 
it is processed. This is a literally a truckload of paper, which travels with the Shuttle in 
wooden “coffins” from building to building. Some of these consist of Problem Reports, 
though most of them are paper records of routine processing. 
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The paper movement at KSC is similar to the SSP Paper management process described 
earlier. At KSC an enormous amount of paper is physically moved through the work 

process system, at a point when most of the essential information already resides in the 
electronic database. 


At KSC there have been experiments to use personal data assistants (PDAs) for filling out 
PRs, rather than paper. Dan Mondshein reported preliminary success, however, the early 
PDAs lacked robustness and became obsolete. Additionally, the use of PDA’s has not 
and was not budgeted into the recumng costs for replacing paper. We have been told that 
there are plans try again perhaps using the Palm PDA platform. 


4.0 Recommendations 


In order to recommend modifications, upgrades, and enhancements to the SSP PRACA 
systems, we must establish two things: first, what PRACA currently is; and second, what 
PRACA should be. This study has endeavored to identify the current state of PRA CA 
(i.e., what PRACA currently is). As for what PRACA should be, there are three fairly 
distinct mental pictures emerging from the team’s interviews with the PRACA 
workforce, SSP management, and NASA senior management. These are: 


1. The. Proiect/Element d omain expert view: 

PRACA should remain a collection of relatively simple databases that support the 
work process and record-keeping functions. These databases are designed primarily 
to support the domain experts who are responsible for reporting Project/Element 
status and trending to the SSP. The domain experts would prefer that the SSP 
continue to rely upon the domain experts for data extraction, filtering, analysis, 
interpretation and reporting from the PRACA databases and other sources. 

2. The Fund Sou^ p r.SSP/Code Ml view: . . . 

PRACA is a multi-center data system that is vital to the SSP mission. The domain 
experts’ role in the PRACA system is consistent with the team problem resolution 
approach and is not seen as a potential problem. Ongoing reviews and relatively 
stable workforce will sustain the system’s viability into the future. Additional work 
on PRACA should be justified based upon new capability. Doing the same thing 
better, faster, or cheaper is not necessarily a high priority for funding. 


3 The NASA Informati on Management view: 

PRACA should be a state-of-the-art data warehouse capable of data mining and 
advanced data analysis and trending using a simple and uniform point and click 
interface. The system should preclude data errors, incomplete problem tracking, an 
catch potential problems that might otherwise go undetected (e.g., “escapes and 
“diving catches”). The system should reduce the sole dependence on domain experts 
and corporate knowledge, placing the power of top-level knowledge and information 
in the hands of anyone with access to the system via a simple user inte rfsce. 
Additionally, advanced data mining capabilities would support the SRQ&MA 
analysts to improve the speed and accuracy of their assessments. With the Shuttle 
expected to fly another 25 years, the system architecture must be dynamic and 
capable of overcoming changes in workforce, technology, and flight rate. The system 
should be enhanced to provide a foundation enabling the future implementation of a 
safety and risk prognostics capability. The system should serve as a model and 
pathfinder for the Agency. 


As we have noted in previous sections of the study, there is no overall SSP PRACA 
owner and vision declaration for scope and functionality. The general vision of capability 
proposed in the NSTS 07700 volume XI is not being fulfilled with the present PRACA 

systems. 



The Study team has chosen to present its recommendations with attention to the 
identification of an owner, the decisions yet to be made, and the assumption that the 
NASA Information Management view of the PRACA system should serve as a goal for 
the final system state and our recommendations. Sensitivity to the Project and Fund 
Source views was maintained but as secondary considerations. Given this attention and 
assumption guideline, this Study has identified several recommendations for 
modification, upgrade, and enhancement of the SSP PRACA System. The 
recommendations are organized as follows: 

1 . A set of general recommendations. 

2. Specific recommendations addressing the four technical assessment areas (UI, 
Database and Data Management, Network and System Architecture, and Work 
Process Study). 

These recommendations are discussed in the following sub-sections. 

4.7 General Recommendations 

"4.1.1 Gtabat -Perspective 


The SSP-identified PRACA owner needs to make a global assessment of PRACA with 
both a short-term and long-term view. It is important to answer vision-defining questions 
such as: 

• Is PRACA sufficient for the SSP needs? If so, for how long? 

• Is PRACA to be a cutting edge information management system? Is it to serve as 
an example for Agency emulation? 

• Is PRACA to look beyond SSP focus to leverage other safety and reporting 
systems? (Aviation Safety Program, Commercial aviation scheduling and 
planning systems, model-based reasoning systems, digitized Shuttle systems, 
other NASA PRACA systems, etc.) 

• What is the evolving role for PRACA looking into the next 25 years? 

• What is the relationship of PRACA to the changing NASA workforce? And how 
does that impact PRACA functionality over time? 

• Is PRACA to be the foundation of a Safety and Risk Prognostics System for the 
SSP? 

It is equally (if not more) important to answer design questions driving the requirements 
such as: 

• Who is the owner of the system? 

• Who are the customers for the system’s data? 

• Who are the users of the system? 

• Who are the managers of the system? 

• What skill level(s) is expected of the owner, customers, users, and managers of 
the system? 

• What is the security level of the data in the system, and what is the desired 
visibility in the community? 
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• How large a dependence on expert knowledge and human interpretation is 

. Is i^pennissible/desirable to use data outside of PRACA (and PCASS) or should 
PRACA be the sole source of data access? 

• What are the roles of the Program office as an owner, user, and a customer of 

PRACA information? 

• What is the role of PRACA at the data collection level? At the Program/Element 
level? 

Once these and other similar questions are answered, the SSP should clearly articulate its 
vision and train and/or inform personnel in all of the levels of the PRACA system. 

4^,2 PRACA as an Element of Safety and Risk 
Prognostics 


One of the unrealized possibilities for the PRACA database systems is as a foundation for 
a Safety and Risk Prognostics capability. Prognostics in this sense go beyond simple data 
trending, to provide a true predictive capability that could greatly enhance the decision- 
making capabilities of the SSP and the safety of the Shuttle. 


Recommendation: 

• Establish a plan for PRACA system evolution that will enable the development of 
a future Safety and Risk Prognostics capability. 


Impact: 

. Improve the breadth and depth of the SRQ&MA analyses performed by the 
experts in a given time frame, as well as ensure the high quality of the PRACA 
data for such analyses to be made. 

• Provide a manager-level overview and quick look assessments of Shuttle safety 

and risk data. , . , „ , . 

_ Enable SSP management to be more proactively involved and up-to-date on 

the performance and safety trade-offs for the Shuttle fleet. 
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Figure 28 - PRACA Data Use Concept 

As noted in the figure above, a technology gap exists in the current PRACA technology 
“pyramid.” This technology gap is currently compensated for by the use of domain 
experts to manually search, interpret, filter, and process the data into knowledge for the 
SSP . This technology gap should be eliminated by: 

1 . Implementing improvements to the PRACA systems in the fundamental 
enhancements areas as shown in the figure 

2. Implementing advanced data access, data mining, and unified user interfaces. 

We believe that an improved and enhanced PRACA System could radically improve the 
breadth and depth of the SRQ&MA analyses performed by the experts in a given time 
frame, as well as ensure the high quality of the PRACA data for such analyses to be 
made. Additionally, the future PRACA System would provide a manager-level overview 
and quick look assessments of Shuttle safety and risk data. This will enable SSP 
management to be more proactively involved and up-to-date on the performance and 

safety trade-offs for the Shuttle fleet. Specific recommendations to do this are addressed 
in section 4.2. 


4.1.3 Update of PRACA Requirements 


Since the delivery of the SIAT report in December 1999 and the initial ARC PRACA 
review in January 2000, the SSP created the PET to reassess and revise the NSTS 08126 
requirements for PRACA. In conjunction with this activity, the USA Integrated PRACA 
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Team has addressed many of the underlying processes and motivations for the various 
PRACA systems under its control. These activities do several important things. 

• They unify some of the reporting requirements. 

• They enhance some of the access requirements. 

. They respond to multiple SSP and USA audits and reviews and address the SIAT 
report concerns and the informal ARC Study comments. 

It is also critically important to answer the design questions driving the requirements, 
from section 4.1.1 that have not been completely addressed by the aforementioned SSP 
and USA activities. 


4.1. 3.1 PRACA Owner* 


The creators or Project-level managers of the PRACA systems do not yet view PRACA 
data as a program-wide resource. An overall “PRACA System” as an organizational or 
technological entity does not exist, and was not required by the NSTS 08126 Revision G 
document. The JSC PET has rewritten and enhanced the NSTS 08126 document to a 
Revision H. Revision H better reflects desired scope and global functions required of the 
SSP’s PRACA system. The Space Shuttle Program Review Control Board is expected to 
approve Revision H in the summer of 2000. 


The SSP has identified a Shuttle Program PRACA Owner and USA has identified an 
internal PRACA owner. The team recommends that these owners take the action to 
declare the vision for the “PRACA System” and its evolution over the next 25 years. The 
vision declaration should create a concrete image in the minds of the PRACA workforce, 
from the data collectors through the SRQ&MA analysts to the SSP management office. 


4.1. 3.2 Program-Level Access 

As we discussed in a previous section of the study, the existing “PRACA System” is not 
sufficient for Program-level data mining and SQ&MA assessment. The SSP current y 
meets the necessary constraint of having enough problem reporting data and insight by 
relying on a set of domain experts possessing extensive knowledge of the Shuttle 
subsystems and the PRACA data, and who have access to additional non-PRACA 
(formal) data. These experts produce consolidated reports and summaries tor the bbf 
office from which the SSP performs its tasks and formulates decisions. This sole 
dependence on expert knowledge and domain experts has shielded the SSP office from 
several PRACA deficiencies, including: 

• PRACA data alone does not provide enough information for Program-level 
trending and data mining applications. 

- There are multiple data sources on maintenance, repair, corrective actions and 
engineering dispositions (CARs, hazard reports, engineering databases, expert 
knowledge, etc.) not included in the PRACA systems (or even PCASS) and 
unavailable to the Program Office. 




• Every PRACA system is unique and designed primarily for Project/Element 
(subsystem domain) use. Program use of the systems and their data are mainly 
handled as design patches to the systems. 

• USA s ADAM is incomplete and unable to act as a data warehouse supporting 

cross database data mining. A proposed KSC7USA WebPCASS based upon the 
existing ADAM structure is currently proposed. This is a good first attempt to 
provide unified PRACA data and some associated information, but needs to go 
much farther. c 

• Generating SRQ&MA reports is an extremely time and labor intensive activity 
requiring specialized knowledge and much massaging and cleaning of PRACA 
data. 

To improve Program-level access to the PRACA data, we recommend that the current 

PRACA system be replaced or significantly upgraded. 

Recommendati on s : 

• Clearly identify (list) the Program-level PRACA tasks from a Program-wide 
perspective 

• Establish requirements for a “PRACA System” that performs SSP level PRACA 
tasks (data retrieval, mining and trending needs). This action should be performed 
without consideration of current PRACA capabilities. 

• Design a PRACA System that satisfies these requirements. 

• Either a) Implement this new system or b) Initiate a modernization activity to 
upgrade the current PRACA systems and designs to satisfy the requirements. 

• Replace or enhance the existing WebPCASS proposal based upon the above 
decisions. 

4.1 .4 PRACA Assessment Areas 
Recommendations 


As a foundation for the new PRACA system design, the team has identified specific 
deficiencies and recommended actions for each of the four assessment areas identified in 
our Study approach. It is important to note however, that we believe the Program-wide 
vision for PRACA (i.e., what PRACA should be”) must precede system technology 
changes. 

With regard to the assessment area of functional capabilities and the upgrades 
recommended, it is our opinion that a PRACA Enhancement Project should 

• Satisfy all the Program Offices task-based requirements (see previous section); 

• Satisfy the Project/Element (subsystem domain) work flow management 
requirements; 

• Meet all NASA data security standards; 

• Increase the user base through ease of access and intuitive user interface; 

• Incorporate expert knowledge capture to assist in correct data interpretation and to 
reduce dependency on human corporate/institutional knowledge; 
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• Simplify system management and support requirements; 

• Integrate with other Shuttle data sources to enable Data Mining in support of Risk 
Assessments; 

• Provide advanced IT capabilities for Trending and Analysis in support of 
SRQ&MA requirements; 

• Provide a migration path to a true safety and risk prognostics capability tor the 
SSP. 


4 . 1.5 User-Interface Recommendations 

In the findings, we noted that all the systems have different user interfaces. 
Recommendations: 

• Create a User interface for querying PRACA data and generating basic and 
advanced reporting and Trend Analysis. 

- Implement a standard GUI across all systems. Use a widely distributed and 
supported web browser as the foundation of this interface. 

— Implement transparency to isolate the user from database-to-database 
navigation. 

- Implement a personalizable User Interface allowing customization of the 
interface to the needs of each User. 

- Provide collaborative capabilities to permit and encourage sharing and quenes 
and analyses. 

- Create data mining and reporting tools to support both the advanced 
SRQ&MA analysts as well as the SSP management level overviews of the 

data. . 

- Implement data mining and reporting tools to support the inspectors and assist 

in the assurance of data quality and integrity entered at the work flow 
management level. 

• Implement standard user access control (security) across systems 

• Require that all Safety and Risk data reports be generated using this system to 
enforce the migration of all necessary data into the PRACA System. 


Impact: 

• Reduced training, development cost and management overhead. 

• Single UID and password provides access to all systems 

• Increased visibility into PRACA data, yielding better error checking, increased 
knowledge base, and better-informed and timely decisions. 

• Eliminates the SSP management sole dependence on external data sources and 
domain experts. All necessary data are migrated into PRACA (from the 
supporting databases and expert knowledge sources), eliminating this long-term 
vulnerability. 

Note: The PRACA documentation states that PCASS has the role of integrating problem 
reporting systems from all Project elements and providing a closed-loop verification and 
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accounting process to assure resolution. In section 4. 1.5.4, it states that the PCASS shall 
compile, formulate, and display trend data to identify changes in hardware and software 
performance, reliability, and supportability, and to define program requirements. This 
implies that ADAM (or the future WebPCASS) will integrate program element trend 
systems, perform analysis, and provide data formatted for management visibility. This is 
not what is currently done or possible in the current ADAM implementation, nor what is 
proposed for WebPCASS. 

4.1.6 Database and Data Management 
Recommendations 

In the findings, we noted that all the systems use relational database products but have 
different database implementations. The depth of documentation and requirements varied 
from system to system but generally needs much improvement. The PRACA systems 
were designed for Project/Element use. Program-level reporting and trending is heavily 
dependent upon additional external databases and expert knowledge. In addition, many 
current PRACA data fields in the databases are considered to contain questionable values 
and have “variable” definitions. This increases the dependency on “experts” to filter and 
interpret the data extracted from the PRACA databases. 

Recommendations : 

• Develop consistent database schema and structure, and common data field naming 
conventions and definitions. 

— Schema and structure should be designed to support SSP reporting, trending 
and data mining applications as well as to support the Project/Element work 
flow management. 

- Schema and structure should be well documented to preclude data 
interpretation errors and reporting errors. 

• Standardize on a common COTS database application. 

- Oracle database is most commonly used in PRACA and would be a good 
choice. 

- Implement standard user authentication across systems. 

• Extend the ADAM data warehouse to include relevant non-PRACA databases. 

- Data field naming (or mapping) should be consistent with the PRACA data 
fields, schema, and structure and should be well documented to preclude data 
interpretation errors and reporting errors. 

• Require that all SRQ&MA reports be generated using these databases to enforce 
the migration of all necessary data into the “PRACA System.” 


Impact: 

• Reduced development cost and database management overhead 

• Enable common queries, data mining, and consistent ease of access to data 

• Increased visibility into PRACA data yielding better error checking, increased 
knowledge base, and better-informed and timely decisions. 
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• Reduced dependence of PRACA System on external data sources and domain 
experts. All necessary data are migrated into a unified “PRACA System (from 
the supporting databases and expert knowledge sources) eliminating this long- 
term vulnerability. 


4.1.7 Security Issues, Network and System 
Architecture Recommendations 

In the findings, we noted that all the PRACA systems are accessible from the Internet but 
that they have different security models and LAN implementations to preclude 
unauthorized system access. As with other aspects of the Project/Element-centnc design, 
the network and security implementations are designed to meet local requirements and 
are not designed from a Program-wide perspective. As a result, system-to-system 
communication is very difficult and because of the firewall model (multiple points of 
authentication from disparate locations) is inconsistent, inflexible and unreliable on a 

daily basis. 

There are conflicting security models and it is not clear why the security is being 
implemented in specific ways or if it is actually working (i.e., is it really preventing 
access to unwelcome users?). For example, the firewall security model simply restncts 
access by computers within the LAN to other computers on that same LAN. Without user 
authentication anyone on the LAN can access any machine on that LAN. Adding a layer 
of machine authentication (trusted client) reduces the connection possibilities to specific 
computers, but still does not control who can access the data (as long as they are on a 
trusted client). Adding user authentication restricts access to authorized users, but then is 
often inconsistent and uncoordinated with the firewall and machine authentication, e 
believe that a secure virtual private network architecture provides a better model for the 
PRACA system-wide network architecture and can provide the reliable seamless access 
while enhancing PRACA System security. 

Recommendations: 

• Identify and establish a security requirements document for the PRACA systems 
and their data. 

• Develop consistent security model for all data, networks, and systems associated 
with the PRACA System. 

- Eliminate unnecessary data filters and network security bottlenecks. 

- Implement standard system authentication and encryption across systems. 

• Standardize on a common network authentication and encryption architecture. 

- Use secure network architecture and create PRACA data links on that 
network. 


Impact: 

• Single sign-on for all PRACA related systems. 

• Reduced development cost, maintenance overhead, and user management. 
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• Improved network and system security and access through Center firewalls. 

• Simplified data access at all levels. 

• Ability to assess security measure adequacy (vis-a-vis requirements) and 
effectiveness (as measured by security policy violations). 


4.1.8 Problem Reporting Work Process 
Recommendations 

Work process assessment is a relatively new technique and its application to the PRACA 
study was limited to two sites. On-site tracking and interview of the KSC PRACA data 
collection, and JSC Orbiter problem reporting closure processes were performed during 
the PRACA work process assessment. The team believes this preliminary assessment of 
two PRACA centers will validate the utility of a work process study for the SSP and 
recommends that more detailed studies of additional PRACA sites be performed in the 
future. 

Recommendations: 

• Extend the work process assessment to include other PRACA sites, including 
Marshall Space Flight Center, Palmdale, and the Huntington Beach Problem 
Analysis Center, and expand the study of JSC and KSC processes to include 
observational as well as interview data. 

— This expanded analysis of work flow and work process should allow 

development of models of improved work processes for the use of PRACA at 
individual sites, and for the transfer and sharing of PRACA information 
between centers and organizations. 

• Re-evaluate the strict hierarchy of problems, based on the tree structure of the 
Shuttle assembly. This hierarchy makes it difficult to document or describe 
problems that result from interactions between components in different 
assemblies or systems. 

• Institute training of technicians and engineers in Program-wide PRACA and what 
kinds of information are being requested and why. 

- Resolve local differences in how different organizations fill out Problem 
Report fields. 

— Resolve differences between organizations in how they categorize problems. 

• Determine why there is so much paper movement, and which of it could better be 
accomplished electronically. 

- Some of the work being done appears to be more easily and accurately done 
by a computer than by a human. 

- Evaluate the potential for electronic transfer of all documents and the ability 
to sign the forms on-line with a password-protected electronic signature. 

• Determine if, as suggested, a measure of organizational accountability is “the 
number of problem reports filed.” 

- If true, this affects the report classification decisions. This would tend to 
create a work climate where reducing the number of Problem Reports filed, by 
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tending to identify a nonconformance as a less significant category, has 
incentive. This would skew the data in the PRACA systems. 


Impact: 

• Reduced data entry errors 

• Improved accountability for data quality, timeliness, and follow-up 

• Reduced paper, paper management, and paper trails 

• Improve PR turn around time 

• Increased visibility into PRACA data yielding better error checking, increased 
knowledge base, and better-informed and timely decisions. 

• Simplified data access at the Program level 



5.0 Conclusion 


The SSP PRACA System is an essential component to enable increased Shuttle safety 
and improved assessment of Shuttle readiness for flight. With the emergence of 
significant growth in the capabilities of Information Technology, the SSP PRACA 
System is poised to take advantage of the increased capabilities these advances provide. 
The SSP is motivated to increase the knowledge extraction capability of PRACA by 
using advanced IT tools for improved ease of access, greater breadth and depth of risk 
assessments, enhanced data quality and integrity, faster data mining and trending, and 
progression towards a true Safety and Risk Prognostic capability. 

This Study has identified several areas where improvements in technology or 
implementation can enable a significant SSP PRACA improvement. In addition, the SSP 
PRACA System enhancement activity is capable of benefiting from other development 
activities such as Design for Safety (DfS) Program technology insertion, leverage from 
Aviation Safety Program developments, and other basic information technology 
enhancements coming from the Intelligent Systems Program. 

We believe that an Agency-wide NASA/Industry team in conjunction with the SSP 
PRACA workforce can bring together the required expertise, knowledge base, and 
advanced IT capabilities necessary to achieve NASA’s Information Management vision 
for PRACA. In so doing, PRACA will remain a critical and vital system, enabling a 
reduction in the risk and improvements in safety while supporting the Space Shuttle 
Program into the next decades. 



Appendices 


> Acronyms Defined 


ADAM 

Advanced Data Acquisition and Management 

ARC 

Ames Research Center 

CAAR 

Corrective Action Assistance Request 

CAR 

Corrective Action Report 

CCAR 

Contractor Corrective Action Report 

CIO 

Chief Information Officer 

CIL 

Critical Items List 

COTS 

Commercial Off The Shelf 

CS 

Civil Service 

DfS 

Design for Safety (Program) 

DR 

Discrepancy Report 

ET 

(Space Shuttle) External Tank 

FMEA 

Failure Modes and Effects Analysis 

GFE 

Government-Furnished Equipment 

GOTS 

Government Off The Shelf 

GSE 

Ground Support Equipment 

EPR 

Interim Problem Report 

IT 

Information Technology 

JGPC 

JSC GFE PRACA Center 

JPC 

JSC PRACA Center 

JSC 

Johnson Space Center 

KSC 

Kennedy Space Center 

LRU 

Line Replaceable Unit 

MSFC 

Marshall Space Flight Center 

NCDI 

Non Conformance Data Interface (re: TAIR Station) 

PAC 

Problem Action Center 

PCASS 

Program Compliance Assurance and Status System 

PCR 

Process Change Request 

PDSS 

PRACA Data Support System 

PEP 

PRACA Enhancement Project 

PET 

PRACA Evaluation Team 

PR 

Problem Report 

PRACA 

Problem Resolution and Corrective Action (System) 

PRT 

Problem Resolution Team 

PSA 

PRACA System Architecture 

RSRM 

(Space Shuttle) Reusable Solid Rocket Motor 

SAIC 

Science Applications International Corporation 

SAM 

System Area Manager 

S&MA 

Safety and Mission Assurance 

SIAT 

Shuttle Independent Assessment Team 

SFOC 

Space Flight Operations Contract 

SPDMS 

Shuttle PRACA Data Management System 

SRB 

(Space Shuttle ) Solid Rocket Booster 

SRQ&MA 

Safety, Reliability, Quality and Mission Assurance 



SRU 

SSP 

SSME 

SSRAD 

TAIR 

TCAR 

TM 

UI 

USA 


Shop Replaceable Unit 
Space Shuttle Program 
Space Shuttle Main Engine 
Shuttle Risk and Reliability Database 
Test Assembly Inspection Record 
Team Corrective Action Report 
Technical Monitor 
User Interface 
United Space Alliance 
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> PEP Study Team 


The Key Personnel assigned to this project, their expertise and contact information are: 


Staff 

Experetise 

Contact 

David Korsmeyer, NASA 

Study Architect and 
Analyst 

dkorsmeyer@mail.arc.nasa.gov 

John Schreiner, NASA 

Study Architect and 
Analyst 

j schreiner @ mail . arc .nasa. gov 

Chris Knight, SAIC 

Technology Assessment 

cknight@ptolemy.arc.nasa.gov 

Alex Shaykevich, SAIC 

Technology Assessment 

shaykevi@ptolemy.arc.nasa.gov 

Louise Chan, SAIC 

Database Assessment 

lchan@mail.arc.nasa.gov 

Charlotte Linde, NASA 

Work Process data 
collection and analysis 

clinde@mail.arc.nasa.gov 

Roxana Wales, SAIC 

Work Process data 
collection and analysis 

rwales@mail.arc.nasa.gov 


Table 9 - ARC PEP Study Team 





> Interviews 


The PEP Team performed multiple phone and on-site interviews of the following key 
personnel: 


11 Phone Interviews 

01/04/00 - 06/31/00 - D. Korsmeyer, J. Schreiner 
PET Telecons and, 

KSC: 

Randall “Randy” Segert — CS, IPAS Replacement Owner 

ra: .dall.segert-l ® ksanasa.gov . 141171 867-8515 or 867-8250, Building: M6-0399, 

Room: 3301A 

Ruth M. Harrison - CS Division Chief 

Ruth.Harrison-l@kmail.ksc.nasa.gov . (407) 861-3958 or 861-3957, Building: 
K6-1096, Room: 6309L 

Michael “Mike” Conroy - CS, Chief Systems Eng Banch 
Michael.Conrov-l@kmail.ksc.nasa.gov . (407) 867-4240 or 867-3526, Building- 
M7-0355, Room: 2132 

Jeffrey “Jeff’ I. Goldberg - USA, SFOC ADAM DB admin 
Jeffrey.Goldberg-l@kmail.ksc.nasa.gov . (407) 799-5911, Building: NSLD2, 
Room: 639 

Caroline Paquette - Boeing, PGOC 

Daniel “Dan” B. Mondshein, USA, Mgr Quality Engineering, Quality Data Info 
Systems 

Daniel.Mondshein-l@kmail.ksc.nasa.gov . (407) 861-0890 or 861-0726, Building- 
K61200E, Room: 1033 

MSFC 

Alex Adams — CS PRACA owner, 

Alex.Adams@msfc.nasa.gov . 

John W. McPherson - Hernandez Engineering, UPRACA team lead 
iohn.w.mcpherson@msfc.nasa.gov . 
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JSC: 

Bob Hesselmeyer, CS, TM for SFOC, (TM task issues) 

robert.h.heselmeverl @isc.nasa.gov , Building: 1, Room 757B, Phone: 281-483- 
1292 

Richard Shelton, USA 


Suzanne Little, USA 
Scott Ferguson, SAIC 


On-Site Interviews 


The following interviews were conducted on-site: 

1/5/00 - 1/6/00 - D. Korsmeyer 
JSC: 

William “Bill” Gerstenmaier, CS, Deputy for Space Shuttle Ground Operations 
william. h. gerstenmaierl@isc.nasa.gov 

Linda J. Ham, CS, PRACA Evaluation Team Lead for Shuttle Program 
linda.j.haml@isc.nasa.gov . 

Susan B. Ahrens, USA, SFOC ADAM team lead 
Susan.B.Ahrens@USAHO.unitedspacealliance.com . 

Suzanne Little, USA, SFOC PDMS (Orbiter PRACA) team lead 
Suzanne.Little@USAHO.unitedspaceal1iance.com . 

Sherry Littlefield 
sherrv.littlefield@sw.boeing.com . 

John P. Mulholland, CS, Owner/sign-off of Orbiter CARs and PRs 
iohn.p.mulhollandl@isc.nasa.gov . 

Roger Boyer, SAIC, Orbiter SR&QA Analysis team lead 

David M. Brown, CS, Code NC - Shuttle SR&QA 
david.m.brownel @jsc.nasa.gov . 

Scott Ferguson, SAIC, GFE PRACA team lead 

Jill Diniz, SAIC, (quit recently) Orb Trend Report Team lead 
iill.l.dinizl @isc.nasa.gov . 

Dave Dyer, CS, GFE Owner 

Dorothy Rasco 


3//8/00 - 3/9/00 - D. Korsmeyer 
JSC: 

Linda J. Ham, CS PRACA Evaluation Team Lead for Shuttle Program 


1inda.j.haml@jsc.nasa.gov . 


Susan B. Ahrens, USA SFOC ADAM team lead 

Susan.B.Ahrens@USAHO.unitedspacealliance.com . 

Suzanne Little, USA, SFOC PDMS (Orbiter PRACA) team lead 
Suzanne. I .ittle@USAHO.unitedspacealli ance.com. 

John Mulholland, CS, Owner/sign-off of Orbiter CARs and PRs 
john.p.mulhollandl@isc.nasa.gov . 

Roger Boyer, SAIC, Orbiter SR&QA Analysis team lead 
Tim Adams 

Scott Ferguson, SAIC, GFE PRACA team lead 

Jill Diniz, SAIC, (quit recently) Orb Trend Report Team lead 
jill.l.dinizl @isc.nasa.gov 

Dave Dyer, CS, GFE Owner 

Dorothy Rasco 


3//21/00 - 3/23/00 - D. Korsmeyer, A. Shaykevich , C. Knight 
MSFC: 

John W. McPherson, Hernandez Engineering, UPRACA team lead 
john.w.mcDherson@msfc.nasa.gov . 

Marissa Wofford, 
marisa.wofford@msfc.nasa.gov . 

Sherman Avans 

KSC: 

Ruth M. Harrison - CS Division Chief 

Ruth Harrison-l@kmail.ksc.nasa.gov . (407) 861-3958 or 861-3957, Building: 
K6-1096, Room: 6309L 

Mike Conroy - CS Chief Systems Eng Banch 

Jeffrey “Jeff’ I. Goldberg - USA, SFOC ADAM DB admin 
Jeffrev.Goldherg-l@kmail.ksc.nasa.gov . (407) 799-5911, Building: NSLD2, 

Room: 639 



Melody Fleming 
Caroline Paquette 
Gary White 
Chip Hooper 
Connie Vondell 
A1 Kinney 

Daniel B. Mondshein, USA, Mgr Quality Engineering, Quality Data Info Systems 
DanieI.Mondshein-l@kmail.ksc.nasa.gov . (407) 861-0890 or 861-0726, Building- 
K61200E, Room: 1033 
(KSC SSP SFOC PRACA system - SPDMS) 

4/26/00 - 4/27/00 - D. Korsmeyer, J. Schreiner 
KSC: 

Daniel B. Mondshein, USA, Mgr Quality Engineering, Quality Data Info Systems 
Daniel.Mondshein-l@kmail.ksc.nasa.gov . (407) 861-0890 or 861-0726, Building- 
K61200E, Room: 1033 

Charles “Chip” P. Hooper (reporting) 

- Barbara Chesee (D.B. Architecture) 

JSC: 

Linda J. Ham, CS, JSC HQ TA (PET Lead) 
linda.j.haml@isc.nasa.gov . 

Jack Boykin, CS, Asst Mgr, Space Shuttler Program, COTR for USA SFOC 

Roger Boyer, SAIC Orbiter SR&QA Analysis team lead 

- Michael Penney (expert analysis) 

Bob Graeber 

- Betsy Dyer 
Miguel Hughes 

- Bruce Rastle 
Mike Penney 
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Susan B. Ahrens, USA SFOC ADAM team lead 
Susan.B.Ahrens@USAHO.unitedspacealiiance.com . 
Margaret Guardia (data analysis tools) 


Art Nolting 


5/15-5/19/00 -R. Wales 
JSC: 

John Mulholland, 
john.p.mulhollandl@jsc.nasa.gov 

Deputy Manager for Operations in the Space Shuttle Vehicle Engineering Office 

Roger Boyer, Manager, Analysis Section Shuttle Safety and Mission Assurance, 
roger.l.boyerl @jsc.nasa.gov, 

- Miguel Hughes, Lead Engineer, mi guel. hughes 1 @isc.nas a.£ov 

- Michael Penney, Shuttle Safety Engineer Michael.i.penney 1 @ j sc.nasa.g ov 

Suzanne Little, USA SFOC PDMS (Orbiter PRACA) team lead, 

(281) 282-4312 600 Gemini, Door 10 

.Sii7.anne.Little@USAHO.unitedspacealliance.com . 

Scott Ferguson, Project Lead in the GFE PRACA office 
k.s.ferguson 1 @isc.nasa.gov 

David Dyer, Project Lead in the GFE PRACA office 
David.W.Dverl @jsc.nasa.gov 


5/23 - 5/25/00 - J. Schreiner 
KSC: 

Tues 5/23/00 

Bonnie Hauge, USA, title unknown 

Bonnie.Hauge-1 @kmail.ksc.nasa.gov . (407) 861-0745, or 861-0263, Building: 
K61200B, Room: 1056A, 

- Andrea Tucker (rept: Rich Harvey), 321-799-5522, ADAM 
Melody Flemming (rept: Margaret Guardia), 321-799-5519, ADAM 
David Humphrey (rept: Rich Harvey), 321-861-5711, Trends 

- J. M. Anderson (rept: Dan West), 321-861-5306, Perf. Assessment 

- Rene’ Berglund, (rept: Dan West), 321-861-5279, Perf. Assessment 


Keith Jones, USA, title unknown 

Keith. Jones- 1 @kmail .ksc.nasa.gov . (407) 861-6709 or 861-0502, 
Building: K61200B 

(PRACA politics, options, CARs vs PRs) 



Suzanne Cunningham, CS, title unknown 

Suzanne.Cunningham-l@kmail.ksc.nasa.gov . (407) 867-7167 or 867-7089, 
Building: M6-0399, Room: 2506E 


JSC: 

Wednesday 5/24/ 00: 

Bob Hesselmeyer, CS, TM for SFOC 
robert.h.heselmeyerl ©isc.nasa.gov 
Building: l,Room757B 
Phone: 281-483-1292 


Thursday 5/25/00: 

Linda J. Ham, CS, JSC HQ TA (PET Lead) 
Building: 1, Room 580D 
linda.j.haml @jsc.nasa.gov 

- Suzanne Little, ORB PRACA 

- Richard Shelton, USA IM 

- Susan Ahrens, USA ADAM 

- John Muholland, SSP 
James Orr, Flight Software 

- Scott Ferguson, GFE PRACA 

- David Dyer, GFE PRACA 


Suzanne Little, USA SFOC PDMS (Orbiter PRACA) team lead, 
(281) 282-4312 600 Gemini, Door 10 
Suzanne.Little@USAHO.unitedspacealliance.com . 

- Ann Blackburn, 281-282-4834, 
ann.l.Blackbum@usahq.unitedspacealliance.com 

- Robert Edmonds, 281-282-6638, 
bob.w.Edwards@usahq.unitedspacealliance.com 
Thuy Tran, (281) 853-1690, Thuy.tran@sw.boeing.com 
Tin Dinh, (281) 853-1563. Tin.k.dinh@sw.hoeing.com 


Richard Shelton,USA IM 


5/23-5/24/00- C. Linde 
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KSC: 

Dan Mondshein, Manager, Quality Engineering, Quality Data Information 
Systems, USA. 


Wendy Amster and Carl Thomson, Inspectors, Quality Data Information Systems, 
USA 

Gwen Gaskin, Coder, Quality Data Information Systems, USA, 

Various personnel at the Vehicle Processing TAIR station 



> Points of Contact 


NASA Primary Points of Contact for PRACA System Management, and Operations are: 


Site 

System 

Point of Contact 

JSC 

- SSP Program 

- Linda J. Ham, CS, JSC HQ TA (PET Lead) 


(PET Lead) 

linda.j.haml @jsc.nasa.eov. 


- PDSS 

- Suzanne Little, USA SFOC PDMS (Orbiter PRACA) 


(Orbiter PRACA) 

team lead 

Suzanne.Little@USAHO.unitedspacealliance.com. 


- GFE PRACA 

- Scott Ferguson, SAIC GFE PRACA team lead 


-ADAM 

- Susan B. Ahrens, USA SFOC ADAM team lead 




KSC 

- SPDMS 

- Daniel “Dan” B. Mondshein, USA, Mgr Quality 
Engineering, 

Quality Data Info Systems 

Daniel. Mondshein- 1 @kmail.ksc.nasa.gov. 

(407) 861-0890 or 861-0726, Building: K61200E, 
Room: 1033 


-ADAM 

- Margaret Guardia 


-IPAS 

- Randall “Randy” Segert - CS, IPAS Replacement Owner 
randall.seeert-l@ksc.nasa.gov. 14071 867-8515 or 
867-8250, Building: M6-0399, Room: 3301 A . 

MSFC 

- RSRM 

- Alex Adams, QS-20 D 


-ET 

Don Whirley, QS-10 


- SSME 

- SRB 

John W. McPherson (HEI) 

ARC 

- PEP Team 

- David Korsmeyer, CS, Lead Variational Designs 



dkorsme ver @ mail .arc .nasa. gov. 
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